dir_fan <- path(tempdir(), "naming-fanout")
fan <- Tool1$new(path = system.file("extdata/tool1_outputs", package = "nemo"))
fan$config$get_table_outputs("table7")
#> [1] "table7main" "table7dp"
fan$run(output_dir = dir_fan, format = "parquet", input_id = "input1")
sort(basename(fan$written_files$outpath))
#> [1] "sampleA_2_tool1_table7dp.parquet" "sampleA_2_tool1_table7main.parquet"
#> [3] "sampleA_tool1_table7dp.parquet" "sampleA_tool1_table7main.parquet"Repeat runs of the same sample produce identical file names in different folders. Below: how tidy outputs avoid overwriting each other, and how to keep runs distinguishable after merging.
Anatomy of a tidy filename
{output_dir}/{prefix}_{tool}_{table}.{ext}-
{output_dir}: flat output directory -
{prefix}: input basename minus the schemapattern(sampleA.tool1.table1.tsv=>sampleA) -
{tool}_{table}: the schema table that matched the file, e.g.tool1_table1; for a file that fans out, one file per declared output instead (see below) -
{ext}: fromformat(parquet, db, tsv, csv or rds)
sampleA.tool1.table1.tsv => sampleA_tool1_table1.parquet.
One file, several tables
A file can fan out into several tables, declared under its schema table’s outputs:. Each output is written as {prefix}_{tool}_{output}: table7 splits into table7main (read counts) and table7dp (long-form depth coverage):
The two table7 files (latest, v1.0.0) share basename sampleA, hence sampleA and sampleA_2, as in the repeated-sample case below.
A repeated sample
Three runs, same input files:
src <- system.file("extdata/tool1/latest", package = "nemo")
dir1 <- path(tempdir(), "naming-demo")
dir_runs <- path(dir1, "runs")
run_ids <- c("run1", "run2", "run3")
for (r in run_ids) {
dest <- dir_create(path(dir_runs, r))
file_copy(dir_ls(src, regexp = "table[12]\\.tsv$"), dest, overwrite = TRUE)
}
dir_tree(dir_runs)
#> /tmp/Rtmp1LhGAq/naming-demo/runs
#> ├── run1
#> │ ├── sampleA.tool1.table1.tsv
#> │ └── sampleA.tool1.table2.tsv
#> ├── run2
#> │ ├── sampleA.tool1.table1.tsv
#> │ └── sampleA.tool1.table2.tsv
#> └── run3
#> ├── sampleA.tool1.table1.tsv
#> └── sampleA.tool1.table2.tsvMode A: one recursive tidy
Point Tool1 at the parent folder. Colliding prefixes get _2/_3:
tool <- Tool1$new(path = dir_runs)
tool$list_files() |>
dplyr::select(path, prefix, parser) |>
dplyr::arrange(parser)
#> # A tibble: 6 × 3
#> path prefix parser
#> <chr> <chr> <chr>
#> 1 /tmp/Rtmp1LhGAq/naming-demo/runs/run1/sampleA.tool1.table1.tsv sampleA table1
#> 2 /tmp/Rtmp1LhGAq/naming-demo/runs/run2/sampleA.tool1.table1.tsv sampleA… table1
#> 3 /tmp/Rtmp1LhGAq/naming-demo/runs/run3/sampleA.tool1.table1.tsv sampleA… table1
#> 4 /tmp/Rtmp1LhGAq/naming-demo/runs/run1/sampleA.tool1.table2.tsv sampleA table2
#> 5 /tmp/Rtmp1LhGAq/naming-demo/runs/run2/sampleA.tool1.table2.tsv sampleA… table2
#> 6 /tmp/Rtmp1LhGAq/naming-demo/runs/run3/sampleA.tool1.table2.tsv sampleA… table2No overwrites:
dir_outA <- dir_create(path(dir1, "outA"))
tool$run(
input_id = "input1",
output_dir = dir_outA,
format = "parquet",
prefix_include = TRUE
)
dir_tree(dir_outA)
#> /tmp/Rtmp1LhGAq/naming-demo/outA
#> ├── metadata_tool1.parquet
#> ├── sampleA_2_tool1_table1.parquet
#> ├── sampleA_2_tool1_table2.parquet
#> ├── sampleA_3_tool1_table1.parquet
#> ├── sampleA_3_tool1_table2.parquet
#> ├── sampleA_tool1_table1.parquet
#> └── sampleA_tool1_table2.parquetprefix_include = TRUE adds an input_prefix column, so runs stay distinct after stacking:
dir_ls(dir_outA, regexp = "tool1_table1\\.parquet") |>
purrr::map(arrow::read_parquet) |>
purrr::list_rbind()
#> # A tibble: 9 × 8
#> input_id input_prefix sample_id chromosome start end metric_y metric_z
#> <chr> <chr> <chr> <chr> <int> <int> <dbl> <dbl>
#> 1 input1 sampleA_2 sampleA chr1 10 50 0.4 0.7
#> 2 input1 sampleA_2 sampleA chr2 100 500 0.5 0.8
#> 3 input1 sampleA_2 sampleA chr3 1000 5000 0.6 0.9
#> 4 input1 sampleA_3 sampleA chr1 10 50 0.4 0.7
#> 5 input1 sampleA_3 sampleA chr2 100 500 0.5 0.8
#> 6 input1 sampleA_3 sampleA chr3 1000 5000 0.6 0.9
#> 7 input1 sampleA sampleA chr1 10 50 0.4 0.7
#> 8 input1 sampleA sampleA chr2 100 500 0.5 0.8
#> 9 input1 sampleA sampleA chr3 1000 5000 0.6 0.9Limit: one shared input_id; the only per-run key is input_prefix.
Mode B: per-run tidy
Tidy each run separately with its own input_id and output subfolder:
dir_outB <- dir_create(path(dir1, "outB"))
for (r in run_ids) {
Tool1$new(path = path(dir_runs, r))$run(
input_id = r,
output_dir = path(dir_outB, r),
format = "parquet",
prefix_include = TRUE
)
}
dir_tree(dir_outB)
#> /tmp/Rtmp1LhGAq/naming-demo/outB
#> ├── run1
#> │ ├── metadata_tool1.parquet
#> │ ├── sampleA_tool1_table1.parquet
#> │ └── sampleA_tool1_table2.parquet
#> ├── run2
#> │ ├── metadata_tool1.parquet
#> │ ├── sampleA_tool1_table1.parquet
#> │ └── sampleA_tool1_table2.parquet
#> └── run3
#> ├── metadata_tool1.parquet
#> ├── sampleA_tool1_table1.parquet
#> └── sampleA_tool1_table2.parquetSame file names per subfolder; input_id separates them once merged:
fs::dir_ls(dir_outB, recurse = TRUE, glob = "*tool1_table1.parquet") |>
purrr::map(arrow::read_parquet) |>
purrr::list_rbind()
#> # A tibble: 9 × 8
#> input_id input_prefix sample_id chromosome start end metric_y metric_z
#> <chr> <chr> <chr> <chr> <int> <int> <dbl> <dbl>
#> 1 run1 sampleA sampleA chr1 10 50 0.4 0.7
#> 2 run1 sampleA sampleA chr2 100 500 0.5 0.8
#> 3 run1 sampleA sampleA chr3 1000 5000 0.6 0.9
#> 4 run2 sampleA sampleA chr1 10 50 0.4 0.7
#> 5 run2 sampleA sampleA chr2 100 500 0.5 0.8
#> 6 run2 sampleA sampleA chr3 1000 5000 0.6 0.9
#> 7 run3 sampleA sampleA chr1 10 50 0.4 0.7
#> 8 run3 sampleA sampleA chr2 100 500 0.5 0.8
#> 9 run3 sampleA sampleA chr3 1000 5000 0.6 0.9For a globally-unique key, set output_id, e.g. to a ULID (CLI: --ulid, via the ulid R package).
Rule of thumb
- On disk:
_2/_3suffixes (Mode A) or per-run folders (Mode B) - After merging:
input_prefix,input_id,output_idcolumns - One recursive tidy is enough =>
prefix_include - Want named runs =>
input_id/output_id