Output Naming

When the same sample is processed more than once (e.g. repeat runs of a tool over different dates), you end up with several sets of raw outputs that share identical file names but live under different parent folders. Here we show:

• how tidy outputs are named so that repeated runs avoid overwriting each other
• how to carry a run identifier into the data so the runs are still distinguishable once the files have been merged into a merged parquet dataset or database.

Anatomy of a tidy filename

Every tidy file is named:

{output_dir}/{prefix}_{tool}_{parser}.{ext}

• {output_dir}: the directory to output the tidy files in a flat structure.
• {prefix}: the input file’s basename with the table’s schema pattern stripped off. For sampleA.tool1.table1.tsv the table1 pattern (\.tool1\.table1\.tsv$) is removed, leaving sampleA.
• {tool}_{parser}: the tool name and the matched table (e.g. tool1_table1).
• {ext}: driven by format (parquet, db, tsv, csv or rds).

So sampleA.tool1.table1.tsv tidied in parquet format becomes sampleA_tool1_table1.parquet.

A repeated sample

Let us simulate three runs of the same sample by copying one tool’s outputs into separate folders. Each run holds the same files:

src <- system.file("extdata/tool1/latest", package = "nemo")
dir1 <- path(tempdir(), "naming-demo")
dir_runs <- path(dir1, "runs")
run_ids <- c("run1", "run2", "run3")

for (r in run_ids) {
  dest <- dir_create(path(dir_runs, r))
  file_copy(dir_ls(src, regexp = "table[12]\\.tsv$"), dest, overwrite = TRUE)
}

dir_tree(dir_runs)
#> /tmp/Rtmpo6kvmF/naming-demo/runs
#> ├── run1
#> │   ├── sampleA.tool1.table1.tsv
#> │   └── sampleA.tool1.table2.tsv
#> ├── run2
#> │   ├── sampleA.tool1.table1.tsv
#> │   └── sampleA.tool1.table2.tsv
#> └── run3
#>     ├── sampleA.tool1.table1.tsv
#>     └── sampleA.tool1.table2.tsv

Mode A: one recursive tidy

Point a single Tool1 at the parent of the run folders. list_files() recurses into all three and, because the basenames collide, appends _2 / _3 to disambiguate the prefixes:

tool <- Tool1$new(path = dir_runs)
tool$list_files() |>
  dplyr::select(bname, parser, prefix)
#> # A tibble: 6 × 3
#>   bname                    parser prefix   
#>   <chr>                    <chr>  <chr>    
#> 1 sampleA.tool1.table1.tsv table1 sampleA  
#> 2 sampleA.tool1.table1.tsv table1 sampleA_2
#> 3 sampleA.tool1.table1.tsv table1 sampleA_3
#> 4 sampleA.tool1.table2.tsv table2 sampleA  
#> 5 sampleA.tool1.table2.tsv table2 sampleA_2
#> 6 sampleA.tool1.table2.tsv table2 sampleA_3

As you can see, the files share a basename but get prefixes sampleA, sampleA_2, sampleA_3. Writing them out, nothing gets overwritten:

dir_outA <- dir_create(path(dir1, "outA"))
tool$run(
  input_id = "input1",
  output_dir = dir_outA,
  format = "parquet",
  prefix_include = TRUE
)

dir_tree(dir_outA)
#> /tmp/Rtmpo6kvmF/naming-demo/outA
#> ├── metadata_tool1.parquet
#> ├── sampleA_2_tool1_table1.parquet
#> ├── sampleA_2_tool1_table2.parquet
#> ├── sampleA_3_tool1_table1.parquet
#> ├── sampleA_3_tool1_table2.parquet
#> ├── sampleA_tool1_table1.parquet
#> └── sampleA_tool1_table2.parquet

The prefix_include option also writes the prefix as an input_prefix column in the output file itself, so the source run survives even after the three files are read and stacked into one table:

dir_ls(dir_outA, regexp = "tool1_table1\\.parquet") |>
  purrr::map(arrow::read_parquet) |>
  purrr::list_rbind()
#> # A tibble: 9 × 8
#>   input_id input_prefix sample_id chromosome start   end metric_y metric_z
#>   <chr>    <chr>        <chr>     <chr>      <int> <int>    <dbl>    <dbl>
#> 1 input1   sampleA_2    sampleA   chr1          10    50      0.4      0.7
#> 2 input1   sampleA_2    sampleA   chr2         100   500      0.5      0.8
#> 3 input1   sampleA_2    sampleA   chr3        1000  5000      0.6      0.9
#> 4 input1   sampleA_3    sampleA   chr1          10    50      0.4      0.7
#> 5 input1   sampleA_3    sampleA   chr2         100   500      0.5      0.8
#> 6 input1   sampleA_3    sampleA   chr3        1000  5000      0.6      0.9
#> 7 input1   sampleA      sampleA   chr1          10    50      0.4      0.7
#> 8 input1   sampleA      sampleA   chr2         100   500      0.5      0.8
#> 9 input1   sampleA      sampleA   chr3        1000  5000      0.6      0.9

Mode A is the simplest way to dump every tidy run into one folder without overwriting each other. Its limit: the tidy call shares a single input_id, so the only per-run key is the filename-derived input_prefix.

Mode B: per-run tidy

When you want a more meaningful run identifier than e.g. sampleA_2, you should tidy each run separately and pass a distinct input_id. Because the output basenames are now identical across runs, write each run to its own output subfolder so they don’t get overwritten:

dir_outB <- dir_create(path(dir1, "outB"))
for (r in run_ids) {
  Tool1$new(path = path(dir_runs, r))$run(
    input_id = r,
    output_dir = path(dir_outB, r),
    format = "parquet",
    prefix_include = TRUE
  )
}

dir_tree(dir_outB)
#> /tmp/Rtmpo6kvmF/naming-demo/outB
#> ├── run1
#> │   ├── metadata_tool1.parquet
#> │   ├── sampleA_tool1_table1.parquet
#> │   └── sampleA_tool1_table2.parquet
#> ├── run2
#> │   ├── metadata_tool1.parquet
#> │   ├── sampleA_tool1_table1.parquet
#> │   └── sampleA_tool1_table2.parquet
#> └── run3
#>     ├── metadata_tool1.parquet
#>     ├── sampleA_tool1_table1.parquet
#>     └── sampleA_tool1_table2.parquet

Note the same file names in each subfolder. The folder separates them on disk, and the input_id column keeps them apart once merged:

fs::dir_ls(dir_outB, recurse = TRUE, glob = "*tool1_table1.parquet") |>
  purrr::map(arrow::read_parquet) |>
  purrr::list_rbind()
#> # A tibble: 9 × 8
#>   input_id input_prefix sample_id chromosome start   end metric_y metric_z
#>   <chr>    <chr>        <chr>     <chr>      <int> <int>    <dbl>    <dbl>
#> 1 run1     sampleA      sampleA   chr1          10    50      0.4      0.7
#> 2 run1     sampleA      sampleA   chr2         100   500      0.5      0.8
#> 3 run1     sampleA      sampleA   chr3        1000  5000      0.6      0.9
#> 4 run2     sampleA      sampleA   chr1          10    50      0.4      0.7
#> 5 run2     sampleA      sampleA   chr2         100   500      0.5      0.8
#> 6 run2     sampleA      sampleA   chr3        1000  5000      0.6      0.9
#> 7 run3     sampleA      sampleA   chr1          10    50      0.4      0.7
#> 8 run3     sampleA      sampleA   chr2         100   500      0.5      0.8
#> 9 run3     sampleA      sampleA   chr3        1000  5000      0.6      0.9

For a globally-unique key with no coordination between runs, use output_id = "<your id>" with e.g. an auto-generated ULID (done automatically with the CLI’s --ulid flag via the ulid R package), which would generate an output_id column alongside input_id.

Rule of thumb

• File names stop repeat runs from overwriting each other on disk automatically via _2/_3 (Mode A), or by writing to per-run folders (Mode B).
• Provenance columns (input_prefix, input_id, output_id) let you tell the runs apart after the files are merged into one parquet dataset or loaded into a database, where the filename is no longer visible.
• Use prefix_include when one recursive tidy is enough
• Use input_id / output_id when each run is better described by an explicitly specified name you can control.