Pipeline-compact #338

sigmafelix · 2024-06-11T20:39:58Z

V 0.3.0
Compact pipeline: feature calculation is tested. Also implemented base learners, but yet untested.
Function restructuring: amadeus functions are removed, meta learner functions are kept for reference
- TODO: Documentation
- TODO: tests
Tests were removed accordingly

- aadt is replaced by nei

- Removed (a-la-)duplicate functions of amadeus - Rd cleaned - stdt dependency remains; soon to be deprecated - DESCRIPTION update: amadeus is listed in Remotes:

- mostly skeletonized until meta learner fitting - TODO: prediction grid covariate computation, status check, etc.

- Meta learner fitting - prediction grid - documentation for pipeline functions

- Base structure is established - To be filled and refactored then be tested

- mermaid chart - package name change in _targets.R: scomps to chopin

- gets adam initialization error - FIXME: the returned tensor has 17 instead of 18 in the paper example

- running (frozen)

- use non-function as function error

- Pipeline main code is split into several files for maintenance - MLP with brulee + tidymodels: dev

- calculation code (customized for geos) - gmted is on hold - Stale codes were removed

- 10 locations on a single day - run okay except for four targets (TODO) in feature calculation

- 10 points, single day: each calculation flow works as expected - temporal field transformation/renaming is required to join data.frames successfully - TODO: scaled example (full data + several months) - TODO: looking at possibility of tar_make_future() deprecation - TODO: shift to mirai for crew compatibility?

- does crew(mirai)+future work?

- TODO: clean space-time feature data.frames for neater layout - FIXME: join tables for space-time features (ie. heterogenous temporal resolution)

- Added family tag in roxygen2 doc - Example errors fixed

- spatiotemporal cv based on anticlust::balanced_clustering - Adjacent initial clusters are paired into predefined number of secondary sets

- generate_cv_index: generation mode and preprocessing - vis_rset: spt visualization

- Added anticlust to DESCRIPTION - vis_rset: added viewing angle as the other argument

- Family reclassified - Lint

- pkgdown: family to keywords

- no whitespace in keywords

- error-on option is changed to "error" to avoid remote package import errors - Example naming error fix

sigmafelix · 2024-06-13T18:41:02Z

@kyle-messier I managed to pass the PR checks by putting most of the new functions into nocov status (for the time being, of course) and changing check-standard.yaml setting to fail only if R CMD check returns an error. I will improve the coverage in the next PR. Thank you.

- "under construction" flag removed - _targets.R and pipeline_base_functions.R in inst/targets are synchronized

kyle-messier · 2024-06-13T19:40:09Z

_targets.R

Any issue with user specific paths here? "/ddn/gs1/home/songi2/r-libs"

I've been using a user library path for additional packages that are not available in the system default library path. I could make a subdirectory in the team project directory or somewhere else then will change the value in _targets.R. I think everyone in the DDN user group grpkpmessier will be able to read the library in /ddn/gs1/home/songi2/r-libs.

.github/workflows/check-standard.yaml

sigmafelix

Each review was commented.

sigmafelix · 2024-06-13T19:48:58Z

_targets.R

I've been using a user library path for additional packages that are not available in the system default library path. I could make a subdirectory in the team project directory or somewhere else then will change the value in _targets.R. I think everyone in the DDN user group grpkpmessier will be able to read the library in /ddn/gs1/home/songi2/r-libs.

.github/workflows/check-standard.yaml

kyle-messier

@sigmafelix Just a few things I noticed as comments.

kyle-messier · 2024-06-13T20:09:38Z

input/tidymodels_stcv_integration.R

Are L10-12 deprecated or still in use?

source("R/manipulate_spacetime_data.R")
source("R/cross_validation.R")
source("input/Rinput/processing_functions/filter_unique_sites.R")
`

I am not seeing R/cross_validation.R

This file is no longer relevant, so I removed the file.

kyle-messier · 2024-06-13T20:32:41Z

tools/generate_test_data.Rmd

@@ -21,7 +21,9 @@ knitr::opts_chunk$set(
 )
 if (!require(NRTAPmodel)) {
  pak::pak("Spatiotemporal-Exposures-and-Toxicology/NRTAPmodel@isong-calc-covars")
+  pak::pak("Spatiotemporal-Exposures-and-Toxicology/amadeus")


Old username GitHub paths?

Fixed the path and cleaned redundant lines.

- Old or irrelevant files removed - Fixed GitHub usernames - TODO: remove tools entirely

sigmafelix · 2024-06-13T21:07:17Z

@kyle-messier Thank you very much for reviewing my PR thoroughly. I think I reflected the comments into the recent push and I will merge it into main.

Insang Song and others added 30 commits January 23, 2024 13:59

More test + calculate_covariate entry

184e3d4

- aadt is replaced by nei

stcv index-rsample compatibility function

b571796

first move

e60e2ed

- Removed (a-la-)duplicate functions of amadeus - Rd cleaned - stdt dependency remains; soon to be deprecated - DESCRIPTION update: amadeus is listed in Remotes:

pipeline in progress (02192024)

3595a96

pipeline update (continued)

70e19e5

pipeline update

76e31ea

- mostly skeletonized until meta learner fitting - TODO: prediction grid covariate computation, status check, etc.

pipeline update

6d82528

- Meta learner fitting - prediction grid - documentation for pipeline functions

pipeline external

62ac2ec

- Base structure is established - To be filled and refactored then be tested

pipeline flowchart

4c68e5d

- mermaid chart - package name change in _targets.R: scomps to chopin

static state of sites

9e245b1

pipeline dev

f7e313c

pytorch to rtorch

64601f9

- gets adam initialization error - FIXME: the returned tensor has 17 instead of 18 in the paper example

code porting update

4a9d117

- running (frozen)

translation in progress

b4612b2

stcnet dev

8ce3cec

- use non-function as function error

torch experiments

7210ad6

CUDA issue + figuring out data-tensor connection

0f33a15

pipeline code reorganization for dev

a010323

- Pipeline main code is split into several files for maintenance - MLP with brulee + tidymodels: dev

pipeline dev

0fe9bf0

pipeline dev continued

3401c42

lint + target update

300982a

transfer bulk geos processing function to pipeline base

b3853ff

pipeline in progress

40a8c46

- calculation code (customized for geos) - gmted is on hold - Stale codes were removed

pipeline in progress

15a1289

pipeline dev: calc_geos refactoring

c93d2bd

pipeline dev

79e9b06

- 10 locations on a single day - run okay except for four targets (TODO) in feature calculation

pipeline dev update

3719ad6

- does crew(mirai)+future work?

pipeline on HPC

4c14594

Pipeline on SLURM

bb94005

- TODO: clean space-time feature data.frames for neater layout - FIXME: join tables for space-time features (ie. heterogenous temporal resolution)

Insang Song added 6 commits June 12, 2024 13:29

edited description

368178e

namespace reexport + .Rbuildignore

0b96365

check dependency

95de0bc

Documentation update

862604d

- Added family tag in roxygen2 doc - Example errors fixed

DESCRIPTION reorganized

e96735a

example fix

06e20f8

sigmafelix reopened this Jun 12, 2024

Insang Song added 11 commits June 12, 2024 23:44

base learner + cv

b117bda

- spatiotemporal cv based on anticlust::balanced_clustering - Adjacent initial clusters are paired into predefined number of secondary sets

CV index generation function patch and vis

a994322

- generate_cv_index: generation mode and preprocessing - vis_rset: spt visualization

a

d0fa822

- Added anticlust to DESCRIPTION - vis_rset: added viewing angle as the other argument

pkgdown config fix + a

37b9e22

- Family reclassified - Lint

doc

1152611

- pkgdown: family to keywords

doc fix (2)

a246ff9

- no whitespace in keywords

nocov + yml typo fix

96c002e

no example run

474867d

yaml modified

1d4c180

- error-on option is changed to "error" to avoid remote package import errors - Example naming error fix

All examples will not run

12398e1

lint

aa96037

sigmafelix requested a review from kyle-messier June 13, 2024 18:38

sigmafelix mentioned this pull request Jun 13, 2024

pkgdown #310

Closed

base learner documentation

526eaa1

- "under construction" flag removed - _targets.R and pipeline_base_functions.R in inst/targets are synchronized

kyle-messier reviewed Jun 13, 2024

View reviewed changes

.github/workflows/check-standard.yaml Outdated Show resolved Hide resolved

relocate legacy targets directory

5fc04e3

sigmafelix commented Jun 13, 2024

View reviewed changes

kyle-messier approved these changes Jun 13, 2024

View reviewed changes

clean path + ./input

ad42e43

- Old or irrelevant files removed - Fixed GitHub usernames - TODO: remove tools entirely

sigmafelix merged commit fdd409c into main Jun 13, 2024
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pipeline-compact #338

Pipeline-compact #338

sigmafelix commented Jun 11, 2024 •

edited

Loading

sigmafelix commented Jun 13, 2024

kyle-messier Jun 13, 2024

sigmafelix Jun 13, 2024

sigmafelix left a comment

sigmafelix Jun 13, 2024

kyle-messier left a comment

kyle-messier Jun 13, 2024

sigmafelix Jun 13, 2024

kyle-messier Jun 13, 2024

sigmafelix Jun 13, 2024

sigmafelix commented Jun 13, 2024

Pipeline-compact #338

Pipeline-compact #338

Conversation

sigmafelix commented Jun 11, 2024 • edited Loading

sigmafelix commented Jun 13, 2024

kyle-messier Jun 13, 2024

Choose a reason for hiding this comment

sigmafelix Jun 13, 2024

Choose a reason for hiding this comment

sigmafelix left a comment

Choose a reason for hiding this comment

sigmafelix Jun 13, 2024

Choose a reason for hiding this comment

kyle-messier left a comment

Choose a reason for hiding this comment

kyle-messier Jun 13, 2024

Choose a reason for hiding this comment

sigmafelix Jun 13, 2024

Choose a reason for hiding this comment

kyle-messier Jun 13, 2024

Choose a reason for hiding this comment

sigmafelix Jun 13, 2024

Choose a reason for hiding this comment

sigmafelix commented Jun 13, 2024

sigmafelix commented Jun 11, 2024 •

edited

Loading