Structure

nemo has three base R6 classes: Config, Tool, Workflow. Child packages (tidywigits, tidydragen) subclass Tool/Workflow with tool-specific parsers and tidiers.

Examples below use the built-in Tool1 and Workflow1.

Config

Reads inst/config/tools/<tool>/schema.yaml and serves versioned raw/tidy schemas from it. See ?Config.

• format
• intro
• tables
• schemas

One tables entry per file type:

tables:
  <table_name>:
    description: '<human-readable description>'
    pattern: "<regex matching the file basename>"
    glob: ["<aws s3 sync --include glob>", '...']
    ftype: '<file type — see below>'
    columns:
      - raw: '<raw column name or positional label>'
        tidy: '<tidy snake_case name>'
        type: 'char | int | float'
        description: '<human-readable description>'
        versions: ['<version1>', '<version2>']
    outputs: # optional: only when the file fans out
      <output_name>:
        description: '<human-readable description>'
        shape: 'exact | subset' # optional, default exact
        columns:
          - tidy: '<tidy snake_case name>'
            type: 'char | int | float'
            description: '<human-readable description>'
            versions: ['<version1>', '<version2>']

• pattern: PCRE regex on the file basename; Tool uses it to discover and classify local files.
• glob: list of aws s3 sync --include patterns on the whole key; wf_s3sync_patterns() collects them to decide what to sync down. Usually the basename pattern with a leading * ("\\.tool1\\.table1\\.tsv$" => ["*.tool1.table1.tsv"]).

ftype picks the parser:

ftype	Description	raw: names	Parser
tsv	Header present; tab-delimited	actual column names	parse_file
csv	Header present; comma-delimited	actual column names	parse_file (delim=,)
tsv-nohead	No header; positional columns	X1, X2, …, XN	parse_file_nohead
tsv-keyvalue	No header; 2 cols (key, value); pivoted wide	actual key values	parse_file_keyvalue

Child packages add ftypes by overriding private$extra_ftypes() (named list of ftype -> function(x, table_name)).

versions lists every tool version a column appears in:

# table1 columns
SampleID:   versions: ['v1.2.3', 'v4.5.6', 'latest']  # present in all versions
metricX:    versions: ['v1.2.3']                        # dropped in v4.5.6
metricY:    versions: ['latest']                        # added in latest

Schema for version V = columns whose versions contain V. schema_guess matches a file’s columns to the right version.

A file that splits into several tables (fan-out) declares one outputs entry per table, with versioned tidy columns (raw only when the output is parsed straight from a section of the file). Its tidy_<table>() returns one tibble per output; Tool checks for exactly the declared outputs, each matching the columns (in order) and types of one schema version. shape: 'subset' allows any subset of columns in any order. Each output is written as <tool>_<output>, e.g. Tool1’s table7 => table7main + table7dp.

tool <- params$tool
workflow <- params$workflow
conf <- nemo::Config$new(tool, pkg = "nemo")
conf

## #--- Config nemo::tool1 ---#
## 
## |var   |value |
## |:-----|:-----|
## |tool  |tool1 |
## |pkg   |nemo  |
## |ntbls |6     |

All tables:

conf$get_patterns() |> knitr::kable(caption = glue("{tool} patterns."))

Tool1 patterns.

name	pattern
table1	.tool1.table1.tsv$
table2	.tool1.table2.tsv$
table3	.tool1.table3.tsv$
table4	.tool1.table4.tsv$
table6	.tool1.table6.csv$
table7	.tool1.table7.tsv$

conf$get_globs() |> knitr::kable(caption = glue("{tool} globs."))

Tool1 globs.

name	glob
table1	*.tool1.table1.tsv
table2	*.tool1.table2.tsv
table3	*.tool1.table3.tsv
table4	*.tool1.table4.tsv
table6	*.tool1.table6.csv
table7	*.tool1.table7.tsv

conf$get_ftypes() |> knitr::kable(caption = glue("{tool} file types."))

Tool1 file types.

name	ftype
table1	tsv
table2	tsv
table3	tsv-keyvalue
table4	tsv-nohead
table6	csv
table7	tsv

conf$get_descriptions() |> knitr::kable(caption = glue("{tool} descriptions."))

Tool1 descriptions.

name	description
table1	Table1 for tool1 (tsv: header present, tab-delimited).
table2	Table2 for tool1 (tsv: header present, tab-delimited).
table3	Table3 for tool1 (tsv-keyvalue: no header, 2 cols, col1=key col2=value).
table4	Table4 for tool1 (tsv-nohead: no header, positional cols X1..XN).
table6	Table6 for tool1 (csv: header present, comma-delimited).
table7	Table7 for tool1 (tsv: one wide row, split into two outputs).

One table:

conf$get_pattern("table1")

## [1] "\\.tool1\\.table1\\.tsv$"

conf$get_ftype("table1")

## [1] "tsv"

conf$get_description("table1")

## [1] "Table1 for tool1 (tsv: header present, tab-delimited)."

Output tables: a fan-out table’s declared outputs, or the table itself:

conf$get_outputs() |> knitr::kable(caption = glue("{tool} outputs."))

Tool1 outputs.

name	table	description
table1	table1	Table1 for tool1 (tsv: header present, tab-delimited).
table2	table2	Table2 for tool1 (tsv: header present, tab-delimited).
table3	table3	Table3 for tool1 (tsv-keyvalue: no header, 2 cols, col1=key col2=value).
table4	table4	Table4 for tool1 (tsv-nohead: no header, positional cols X1..XN).
table6	table6	Table6 for tool1 (csv: header present, comma-delimited).
table7main	table7	Table7 read counts, one row per sample.
table7dp	table7	Table7 fraction of bases at each depth threshold, long-form.

conf$get_table_outputs("table7")

## [1] "table7main" "table7dp"

conf$get_output_shape("table7dp")

## [1] "exact"

All versions. Outputs get their own rows, with parent set to their file table; outputs without raw names (table7main, table7dp) are left out of the raw schemas:

conf$get_schemas_raw() |> dplyr::select("name", "parent", "version", "schema")

## # A tibble: 13 × 4
##    name   parent version schema          
##    <chr>  <chr>  <chr>   <list>          
##  1 table1 NA     v1.2.3  <tibble [5 × 2]>
##  2 table1 NA     v4.5.6  <tibble [4 × 2]>
##  3 table1 NA     latest  <tibble [6 × 2]>
##  4 table2 NA     v1.0.0  <tibble [2 × 2]>
##  5 table2 NA     latest  <tibble [3 × 2]>
##  6 table3 NA     v1.0.0  <tibble [3 × 2]>
##  7 table3 NA     latest  <tibble [5 × 2]>
##  8 table4 NA     v1.0.0  <tibble [3 × 2]>
##  9 table4 NA     latest  <tibble [5 × 2]>
## 10 table6 NA     v1.0.0  <tibble [3 × 2]>
## 11 table6 NA     latest  <tibble [4 × 2]>
## 12 table7 NA     v1.0.0  <tibble [5 × 2]>
## 13 table7 NA     latest  <tibble [6 × 2]>

conf$get_schemas_tidy() |> dplyr::select("name", "parent", "version", "schema")

## # A tibble: 17 × 4
##    name       parent version schema          
##    <chr>      <chr>  <chr>   <list>          
##  1 table1     NA     v1.2.3  <tibble [5 × 2]>
##  2 table1     NA     v4.5.6  <tibble [4 × 2]>
##  3 table1     NA     latest  <tibble [6 × 2]>
##  4 table2     NA     v1.0.0  <tibble [2 × 2]>
##  5 table2     NA     latest  <tibble [3 × 2]>
##  6 table3     NA     v1.0.0  <tibble [3 × 2]>
##  7 table3     NA     latest  <tibble [5 × 2]>
##  8 table4     NA     v1.0.0  <tibble [3 × 2]>
##  9 table4     NA     latest  <tibble [5 × 2]>
## 10 table6     NA     v1.0.0  <tibble [3 × 2]>
## 11 table6     NA     latest  <tibble [4 × 2]>
## 12 table7     NA     v1.0.0  <tibble [5 × 2]>
## 13 table7     NA     latest  <tibble [6 × 2]>
## 14 table7main table7 v1.0.0  <tibble [2 × 2]>
## 15 table7main table7 latest  <tibble [3 × 2]>
## 16 table7dp   table7 v1.0.0  <tibble [3 × 2]>
## 17 table7dp   table7 latest  <tibble [3 × 2]>

conf$get_schemas_both()

## # A tibble: 17 × 5
##    name       tbl_description                                                version schema   parent
##    <chr>      <chr>                                                          <chr>   <list>   <chr> 
##  1 table1     Table1 for tool1 (tsv: header present, tab-delimited).         v1.2.3  <tibble> NA    
##  2 table1     Table1 for tool1 (tsv: header present, tab-delimited).         v4.5.6  <tibble> NA    
##  3 table1     Table1 for tool1 (tsv: header present, tab-delimited).         latest  <tibble> NA    
##  4 table2     Table2 for tool1 (tsv: header present, tab-delimited).         v1.0.0  <tibble> NA    
##  5 table2     Table2 for tool1 (tsv: header present, tab-delimited).         latest  <tibble> NA    
##  6 table3     Table3 for tool1 (tsv-keyvalue: no header, 2 cols, col1=key c… v1.0.0  <tibble> NA    
##  7 table3     Table3 for tool1 (tsv-keyvalue: no header, 2 cols, col1=key c… latest  <tibble> NA    
##  8 table4     Table4 for tool1 (tsv-nohead: no header, positional cols X1..… v1.0.0  <tibble> NA    
##  9 table4     Table4 for tool1 (tsv-nohead: no header, positional cols X1..… latest  <tibble> NA    
## 10 table6     Table6 for tool1 (csv: header present, comma-delimited).       v1.0.0  <tibble> NA    
## 11 table6     Table6 for tool1 (csv: header present, comma-delimited).       latest  <tibble> NA    
## 12 table7     Table7 for tool1 (tsv: one wide row, split into two outputs).  v1.0.0  <tibble> NA    
## 13 table7     Table7 for tool1 (tsv: one wide row, split into two outputs).  latest  <tibble> NA    
## 14 table7main Table7 read counts, one row per sample.                        v1.0.0  <tibble> table7
## 15 table7main Table7 read counts, one row per sample.                        latest  <tibble> table7
## 16 table7dp   Table7 fraction of bases at each depth threshold, long-form.   v1.0.0  <tibble> table7
## 17 table7dp   Table7 fraction of bases at each depth threshold, long-form.   latest  <tibble> table7

One table, one version:

conf$get_schema_raw("table1", version = "v1.2.3")

## # A tibble: 5 × 3
##   version field      type 
##   <chr>   <chr>      <chr>
## 1 v1.2.3  SampleID   c    
## 2 v1.2.3  Chromosome c    
## 3 v1.2.3  Start      i    
## 4 v1.2.3  End        i    
## 5 v1.2.3  metricX    d

conf$get_schema_tidy("table1", version = "v1.2.3")

## # A tibble: 5 × 3
##   version field      type 
##   <chr>   <chr>      <chr>
## 1 v1.2.3  sample_id  c    
## 2 v1.2.3  chromosome c    
## 3 v1.2.3  start      i    
## 4 v1.2.3  end        i    
## 5 v1.2.3  metric_x   d

Raw-to-tidy column map, for use in tidy_* methods:

conf$get_col_map("table6")

## # A tibble: 4 × 4
##   raw      tidy      type  description    
##   <chr>    <chr>     <chr> <chr>          
## 1 GeneId   gene_id   c     Ensembl gene ID
## 2 GeneName gene_name c     gene name      
## 3 AdjTPM   tpm_adj   d     adjusted TPM   
## 4 RawTPM   tpm_raw   d     raw TPM

Tool

Finds, parses, tidies and writes one tool’s files. See ?Tool.

• initialise
• config
• list
• tidy
• fan-out
• filter
• write
• run

tool1_path <- system.file("extdata/tool1", package = "nemo")
t1 <- nemo::Tool1$new(path = tool1_path)
t1

## #--- Tool nemo::tool1 ---#
## 
## |var     |value                                                                     |
## |:-------|:-------------------------------------------------------------------------|
## |name    |tool1                                                                     |
## |path    |/home/runner/miniconda3/envs/pkgdown_env/lib/R/library/nemo/extdata/tool1 |
## |files   |11                                                                        |
## |tidied  |false                                                                     |
## |written |false                                                                     |

t1$config

## #--- Config nemo::tool1 ---#
## 
## |var   |value |
## |:-----|:-----|
## |tool  |tool1 |
## |pkg   |nemo  |
## |ntbls |6     |

t1$config$get_patterns()

## # A tibble: 6 × 2
##   name   pattern                   
##   <chr>  <chr>                     
## 1 table1 "\\.tool1\\.table1\\.tsv$"
## 2 table2 "\\.tool1\\.table2\\.tsv$"
## 3 table3 "\\.tool1\\.table3\\.tsv$"
## 4 table4 "\\.tool1\\.table4\\.tsv$"
## 5 table6 "\\.tool1\\.table6\\.csv$"
## 6 table7 "\\.tool1\\.table7\\.tsv$"

t1$config$get_schema_raw("table1", version = "v1.2.3")

## # A tibble: 5 × 3
##   version field      type 
##   <chr>   <chr>      <chr>
## 1 v1.2.3  SampleID   c    
## 2 v1.2.3  Chromosome c    
## 3 v1.2.3  Start      i    
## 4 v1.2.3  End        i    
## 5 v1.2.3  metricX    d

# t1$config$get_schema_raw("table1", version = "latest") # default
t1$config$get_schema_tidy("table1", version = "v1.2.3")

## # A tibble: 5 × 3
##   version field      type 
##   <chr>   <chr>      <chr>
## 1 v1.2.3  sample_id  c    
## 2 v1.2.3  chromosome c    
## 3 v1.2.3  start      i    
## 4 v1.2.3  end        i    
## 5 v1.2.3  metric_x   d

# t1$config$get_schema_tidy("table1", version = "latest") # default

lf <- t1$list_files()
lf |> dplyr::slice(1) |> str()

## tibble [1 × 9] (S3: tbl_df/tbl/data.frame)
##  $ tool_parser  : chr "tool1_table1"
##  $ parser       : chr "table1"
##  $ bname        : chr "sampleA.tool1.table1.tsv"
##  $ size         : 'fs_bytes' num 133
##  $ lastmodified : POSIXct[1:1], format: "2026-11-09 07:10:24"
##  $ path         : chr "/home/runner/miniconda3/envs/pkgdown_env/lib/R/library/nemo/extdata/tool1/latest/sampleA.tool1.table1.tsv"
##  $ pattern      : chr "\\.tool1\\.table1\\.tsv$"
##  $ prefix       : chr "sampleA"
##  $ prefix_suffix: chr ""

tidy() modifies the object in place, storing the tidy (and optionally raw) tibbles. Tidy tibbles are nested, allowing several per file:

t1$tidy(keep_raw = TRUE)
tbls <- t1$get_tbls()
tbls

## # A tibble: 11 × 11
##    tool_parser  parser bname    size lastmodified        path  pattern prefix prefix_suffix raw     
##    <chr>        <chr>  <chr>   <fs:> <dttm>              <chr> <chr>   <chr>  <chr>         <list>  
##  1 tool1_table1 table1 sample…   133 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""            <tibble>
##  2 tool1_table2 table2 sample…    70 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""            <tibble>
##  3 tool1_table3 table3 sample…    83 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""            <tibble>
##  4 tool1_table4 table4 sample…    52 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""            <tibble>
##  5 tool1_table6 table6 sample…   100 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""            <tibble>
##  6 tool1_table2 table2 sample…    47 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"          <tibble>
##  7 tool1_table3 table3 sample…    48 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"          <tibble>
##  8 tool1_table4 table4 sample…    34 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"          <tibble>
##  9 tool1_table6 table6 sample…    78 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"          <tibble>
## 10 tool1_table1 table1 sample…   113 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"          <tibble>
## 11 tool1_table1 table1 sample…    93 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_3"          <tibble>
## # ℹ 1 more variable: tidy <list>

tbls$raw[[1]] |> dplyr::glimpse()

## Rows: 3
## Columns: 6
## $ SampleID   <chr> "sampleA", "sampleA", "sampleA"
## $ Chromosome <chr> "chr1", "chr2", "chr3"
## $ Start      <int> 10, 100, 1000
## $ End        <int> 50, 500, 5000
## $ metricY    <dbl> 0.4, 0.5, 0.6
## $ metricZ    <dbl> 0.7, 0.8, 0.9

tbls$tidy[[1]][["data"]][[1]] |> dplyr::glimpse()

## Rows: 3
## Columns: 6
## $ sample_id  <chr> "sampleA", "sampleA", "sampleA"
## $ chromosome <chr> "chr1", "chr2", "chr3"
## $ start      <int> 10, 100, 1000
## $ end        <int> 50, 500, 5000
## $ metric_y   <dbl> 0.4, 0.5, 0.6
## $ metric_z   <dbl> 0.7, 0.8, 0.9

table7 (in extdata/tool1_outputs) splits into its two schema outputs. The tidy list-column holds one tibble per output:

t7 <- nemo::Tool1$new(
  path = system.file("extdata/tool1_outputs", package = "nemo")
)
t7$tidy()
t7$get_tbls()$tidy[[1]]

## # A tibble: 2 × 2
##   name       data            
##   <chr>      <list>          
## 1 table7main <tibble [1 × 3]>
## 2 table7dp   <tibble [3 × 3]>

t7$get_tbls()$tidy[[1]][["data"]][[2]]

## # A tibble: 3 × 3
##   sample_id    dp   pct
##   <chr>     <dbl> <dbl>
## 1 sampleA      10  0.98
## 2 sampleA      20  0.9 
## 3 sampleA      30  0.75

Keep/drop tool_parsers with include/exclude:

t2 <- nemo::Tool1$new(path = tool1_path)
t2$list_files()

## # A tibble: 11 × 9
##    tool_parser  parser bname             size lastmodified        path  pattern prefix prefix_suffix
##    <chr>        <chr>  <chr>            <fs:> <dttm>              <chr> <chr>   <chr>  <chr>        
##  1 tool1_table1 table1 sampleA.tool1.t…   133 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""           
##  2 tool1_table2 table2 sampleA.tool1.t…    70 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""           
##  3 tool1_table3 table3 sampleA.tool1.t…    83 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""           
##  4 tool1_table4 table4 sampleA.tool1.t…    52 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""           
##  5 tool1_table6 table6 sampleA.tool1.t…   100 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""           
##  6 tool1_table2 table2 sampleA.tool1.t…    47 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"         
##  7 tool1_table3 table3 sampleA.tool1.t…    48 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"         
##  8 tool1_table4 table4 sampleA.tool1.t…    34 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"         
##  9 tool1_table6 table6 sampleA.tool1.t…    78 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"         
## 10 tool1_table1 table1 sampleA.tool1.t…   113 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"         
## 11 tool1_table1 table1 sampleA.tool1.t…    93 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_3"

t2$filter_files(include = c("tool1_table2", "tool1_table3"))
t2$list_files()

## # A tibble: 4 × 9
##   tool_parser  parser bname              size lastmodified        path  pattern prefix prefix_suffix
##   <chr>        <chr>  <chr>             <fs:> <dttm>              <chr> <chr>   <chr>  <chr>        
## 1 tool1_table2 table2 sampleA.tool1.ta…    70 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""           
## 2 tool1_table3 table3 sampleA.tool1.ta…    83 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""           
## 3 tool1_table2 table2 sampleA.tool1.ta…    47 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"         
## 4 tool1_table3 table3 sampleA.tool1.ta…    48 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"

Formats: parquet, db, tsv, csv, rds. Optional input_id, output_id and input_prefix (prefix_include = TRUE) columns get prepended. See output naming.

t2$tidy() # first need to tidy
outdir1 <- file.path(tempdir(), "tool1_write")
fmt <- "csv"
t2$write(
  output_dir = outdir1,
  format = fmt,
  input_id = "run123",
  output_id = "out123",
  prefix_include = TRUE
)
wfiles <- fs::dir_info(outdir1) |>
  dplyr::mutate(bname = basename(.data$path)) |>
  dplyr::select("bname", "size", "type")
wfiles

## # A tibble: 5 × 3
##   bname                                size type 
##   <chr>                         <fs::bytes> <fct>
## 1 metadata_tool1.parquet              4.71K file 
## 2 sampleA_2_tool1_table2.csv.gz          83 file 
## 3 sampleA_2_tool1_table3.csv.gz          98 file 
## 4 sampleA_tool1_table2.csv.gz            99 file 
## 5 sampleA_tool1_table3.csv.gz           111 file

run() = filter + tidy + write:

t3 <- nemo::Tool1$new(path = tool1_path)
outdir2 <- file.path(tempdir(), "t3")
t3$list_files()

## # A tibble: 11 × 9
##    tool_parser  parser bname             size lastmodified        path  pattern prefix prefix_suffix
##    <chr>        <chr>  <chr>            <fs:> <dttm>              <chr> <chr>   <chr>  <chr>        
##  1 tool1_table1 table1 sampleA.tool1.t…   133 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""           
##  2 tool1_table2 table2 sampleA.tool1.t…    70 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""           
##  3 tool1_table3 table3 sampleA.tool1.t…    83 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""           
##  4 tool1_table4 table4 sampleA.tool1.t…    52 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""           
##  5 tool1_table6 table6 sampleA.tool1.t…   100 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""           
##  6 tool1_table2 table2 sampleA.tool1.t…    47 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"         
##  7 tool1_table3 table3 sampleA.tool1.t…    48 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"         
##  8 tool1_table4 table4 sampleA.tool1.t…    34 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"         
##  9 tool1_table6 table6 sampleA.tool1.t…    78 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"         
## 10 tool1_table1 table1 sampleA.tool1.t…   113 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"         
## 11 tool1_table1 table1 sampleA.tool1.t…    93 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_3"

t3$run(
  output_dir = outdir2,
  format = "tsv",
  input_id = "run_t3",
  output_id = "out456",
  prefix_include = TRUE
)
wfiles2 <- fs::dir_info(outdir2) |>
  dplyr::mutate(bname = basename(.data$path)) |>
  dplyr::select("path", "bname", "size", "type")
wfiles2 |> dplyr::select(-"path")

## # A tibble: 12 × 3
##    bname                                size type 
##    <chr>                         <fs::bytes> <fct>
##  1 metadata_tool1.parquet              4.83K file 
##  2 sampleA_2_tool1_table1.tsv.gz         127 file 
##  3 sampleA_2_tool1_table2.tsv.gz          86 file 
##  4 sampleA_2_tool1_table3.tsv.gz         100 file 
##  5 sampleA_2_tool1_table4.tsv.gz         103 file 
##  6 sampleA_2_tool1_table6.tsv.gz         129 file 
##  7 sampleA_3_tool1_table1.tsv.gz         112 file 
##  8 sampleA_tool1_table1.tsv.gz           137 file 
##  9 sampleA_tool1_table2.tsv.gz           102 file 
## 10 sampleA_tool1_table3.tsv.gz           112 file 
## 11 sampleA_tool1_table4.tsv.gz           122 file 
## 12 sampleA_tool1_table6.tsv.gz           142 file

tbl1_path <- wfiles2 |>
  dplyr::filter(stringr::str_detect(.data$bname, "tool1_table1")) |>
  dplyr::pull("path") |>
  head(1)
readr::read_tsv(tbl1_path, show_col_types = FALSE)

## # A tibble: 3 × 8
##   input_id input_prefix output_id sample_id chromosome start   end metric_x
##   <chr>    <chr>        <chr>     <chr>     <chr>      <dbl> <dbl>    <dbl>
## 1 run_t3   sampleA_2    out456    sampleA   chr1          10    50      0.1
## 2 run_t3   sampleA_2    out456    sampleA   chr2         100   500      0.2
## 3 run_t3   sampleA_2    out456    sampleA   chr3        1000  5000      0.3

Workflow

A list of Tools with the same list_files/filter_files/tidy/write/run API, plus get_tools(). Workflow1 wraps a single Tool1. See ?Workflow.

w <- system.file("extdata/tool1", package = "nemo") |>
  nemo::Workflow1$new()
outdir3 <- file.path(tempdir(), "w1")
w$get_tools()

## $tool1
## #--- Tool nemo::tool1 ---#
## 
## |var     |value     |
## |:-------|:---------|
## |name    |tool1     |
## |path    |<ignored> |
## |files   |11        |
## |tidied  |false     |
## |written |false     |

w$list_files()

## # A tibble: 11 × 10
##    tool  tool_parser  parser bname       size lastmodified        path  pattern prefix prefix_suffix
##    <chr> <chr>        <chr>  <chr>      <fs:> <dttm>              <chr> <chr>   <chr>  <chr>        
##  1 tool1 tool1_table1 table1 sampleA.t…   133 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""           
##  2 tool1 tool1_table2 table2 sampleA.t…    70 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""           
##  3 tool1 tool1_table3 table3 sampleA.t…    83 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""           
##  4 tool1 tool1_table4 table4 sampleA.t…    52 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""           
##  5 tool1 tool1_table6 table6 sampleA.t…   100 2026-11-09 07:10:24 /hom… "\\.to… sampl… ""           
##  6 tool1 tool1_table2 table2 sampleA.t…    47 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"         
##  7 tool1 tool1_table3 table3 sampleA.t…    48 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"         
##  8 tool1 tool1_table4 table4 sampleA.t…    34 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"         
##  9 tool1 tool1_table6 table6 sampleA.t…    78 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"         
## 10 tool1 tool1_table1 table1 sampleA.t…   113 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_2"         
## 11 tool1 tool1_table1 table1 sampleA.t…    93 2026-11-09 07:10:24 /hom… "\\.to… sampl… "_3"

x <- w$run(
  output_dir = outdir3,
  format = "tsv",
  input_id = "wf1_run1",
  output_id = "out1",
  prefix_include = TRUE
)
wfiles3 <- fs::dir_info(outdir3) |>
  dplyr::select(1:5) |>
  dplyr::mutate(bname = basename(.data$path))
wfiles3 |>
  dplyr::select("bname", "size", "type")

## # A tibble: 12 × 3
##    bname                                size type 
##    <chr>                         <fs::bytes> <fct>
##  1 metadata.parquet                    4.83K file 
##  2 sampleA_2_tool1_table1.tsv.gz         126 file 
##  3 sampleA_2_tool1_table2.tsv.gz          85 file 
##  4 sampleA_2_tool1_table3.tsv.gz          99 file 
##  5 sampleA_2_tool1_table4.tsv.gz         102 file 
##  6 sampleA_2_tool1_table6.tsv.gz         129 file 
##  7 sampleA_3_tool1_table1.tsv.gz         112 file 
##  8 sampleA_tool1_table1.tsv.gz           137 file 
##  9 sampleA_tool1_table2.tsv.gz           101 file 
## 10 sampleA_tool1_table3.tsv.gz           112 file 
## 11 sampleA_tool1_table4.tsv.gz           121 file 
## 12 sampleA_tool1_table6.tsv.gz           142 file

Metadata

Workflow$write()/$run() (non-db formats) also write a single-row metadata.parquet: IDs, paths, pkg versions, files written.

meta <- arrow::read_parquet(file.path(outdir3, "metadata.parquet"))
meta |> dplyr::glimpse()

## Rows: 1
## Columns: 6
## $ input_id     <chr> "wf1_run1"
## $ output_id    <chr> "out1"
## $ input_dirs   <list<character>> "/home/runner/miniconda3/envs/pkgdown_env/lib/R/library/nemo/extdata/tool1"…
## $ output_dir   <chr> "/tmp/RtmpMaVS62/w1"
## $ pkg_versions <list<
##   tbl_df<
##     name   : character
##     version: character
##   >
## >> [<tbl_df[1 x 2]>]
## $ files        <list<
##   tbl_df<
##     tbl   : character
##     prefix: character
##     fout  : character
##     fin   : character
##   >
## >> [<tbl_df[11 x 4]>]…

Unnest list columns:

meta |> dplyr::select("pkg_versions") |> tidyr::unnest("pkg_versions")

## # A tibble: 1 × 2
##   name  version
##   <chr> <chr>  
## 1 nemo  0.2.2

meta |> dplyr::select("files") |> tidyr::unnest("files")

## # A tibble: 11 × 4
##    tbl          prefix    fout                          fin                                         
##    <chr>        <chr>     <chr>                         <chr>                                       
##  1 tool1_table1 sampleA   sampleA_tool1_table1.tsv.gz   /home/runner/miniconda3/envs/pkgdown_env/li…
##  2 tool1_table2 sampleA   sampleA_tool1_table2.tsv.gz   /home/runner/miniconda3/envs/pkgdown_env/li…
##  3 tool1_table3 sampleA   sampleA_tool1_table3.tsv.gz   /home/runner/miniconda3/envs/pkgdown_env/li…
##  4 tool1_table4 sampleA   sampleA_tool1_table4.tsv.gz   /home/runner/miniconda3/envs/pkgdown_env/li…
##  5 tool1_table6 sampleA   sampleA_tool1_table6.tsv.gz   /home/runner/miniconda3/envs/pkgdown_env/li…
##  6 tool1_table2 sampleA_2 sampleA_2_tool1_table2.tsv.gz /home/runner/miniconda3/envs/pkgdown_env/li…
##  7 tool1_table3 sampleA_2 sampleA_2_tool1_table3.tsv.gz /home/runner/miniconda3/envs/pkgdown_env/li…
##  8 tool1_table4 sampleA_2 sampleA_2_tool1_table4.tsv.gz /home/runner/miniconda3/envs/pkgdown_env/li…
##  9 tool1_table6 sampleA_2 sampleA_2_tool1_table6.tsv.gz /home/runner/miniconda3/envs/pkgdown_env/li…
## 10 tool1_table1 sampleA_2 sampleA_2_tool1_table1.tsv.gz /home/runner/miniconda3/envs/pkgdown_env/li…
## 11 tool1_table1 sampleA_3 sampleA_3_tool1_table1.tsv.gz /home/runner/miniconda3/envs/pkgdown_env/li…