Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Azure Batch configurable startTask #4841

Closed
wants to merge 30 commits into from
Closed

Azure Batch configurable startTask #4841

wants to merge 30 commits into from

Commits on Mar 22, 2024

  1. Fix Azure Batch containerConfiguration which needs Docker type set pe… 

    …rmanently

Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
    @adamrtalbot
    adamrtalbot committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    da3a436 View commit details
    Browse the repository at this point in the history

  2. Azure Batch: Configurable startTask per pool 

    Additions:
 - Make Azure Batch pool startTask configurable.
 - Can be used to modifiy the existing worker node. Use cases:
    - Install more recent version of AzCopy
    - Install monitoring software like Azure Insights
    - Use NVME SSDs for storage
    - Flash Dragen firmware to FPGA chips

Pool specific so you can differentiate between different hardware requirements.

Closes nextflow-io#4259

Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
    @adamrtalbot
    adamrtalbot committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    413ea8d View commit details
    Browse the repository at this point in the history

  3. Bump azure plugin version in docs 

    Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
    @adamrtalbot
    adamrtalbot committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    d714263 View commit details
    Browse the repository at this point in the history

  4. comma 

    Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
    @adamrtalbot
    adamrtalbot committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    59d0b31 View commit details
    Browse the repository at this point in the history

  5. Move startTasks docs below registry and vnet because it seems less us… 

    …eful than them

Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
    @adamrtalbot
    adamrtalbot committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    a99e616 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'master' into azure_batch_configurable_start_task

    @adamrtalbot
    adamrtalbot authored Mar 22, 2024
    Configuration menu
    Copy the full SHA
    baaac49 View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2024

  1. Merge branch 'master' into azure_batch_configurable_start_task

    @adamrtalbot
    adamrtalbot authored Mar 25, 2024
    Configuration menu
    Copy the full SHA
    e0bf9f7 View commit details
    Browse the repository at this point in the history

Commits on Mar 27, 2024

  1. Merge branch 'master' into azure_batch_configurable_start_task 

    Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
    @adamrtalbot
    adamrtalbot authored Mar 27, 2024
    Configuration menu
    Copy the full SHA
    16476f8 View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2024

  1. Merge branch 'master' into azure_batch_configurable_start_task

    @bentsherman
    bentsherman authored Mar 28, 2024
    Configuration menu
    Copy the full SHA
    ad4f2a1 View commit details
    Browse the repository at this point in the history

Commits on Apr 3, 2024

  1. Fix test when missing google secret 

    Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
    @pditommaso
    pditommaso committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    be15693 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into azure_batch_configurable_start_task

    @bentsherman
    bentsherman authored Apr 3, 2024
    Configuration menu
    Copy the full SHA
    8ae8eae View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into azure_batch_configurable_start_task 

    Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
    @pditommaso
    pditommaso authored Apr 3, 2024
    Configuration menu
    Copy the full SHA
    119c3ca View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2024

  1. Merge branch 'master' into azure_batch_configurable_start_task

    @adamrtalbot
    adamrtalbot committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    b42d983 View commit details
    Browse the repository at this point in the history

  2. Combine azcopy and custom start task based on options 

    Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
    @adamrtalbot
    adamrtalbot committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    33969d3 View commit details
    Browse the repository at this point in the history

  3. Remove old code. 

    Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
    @adamrtalbot
    adamrtalbot authored Apr 5, 2024
    Configuration menu
    Copy the full SHA
    0abaf47 View commit details
    Browse the repository at this point in the history

  4. Prevent startTask being added to Azure Batch pool if not configured 

    Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
    @adamrtalbot
    adamrtalbot committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    7c2af6a View commit details
    Browse the repository at this point in the history

  5. Add tests for Azure Batch createStartTask method 

    Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
    @adamrtalbot
    adamrtalbot committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    cd416e4 View commit details
    Browse the repository at this point in the history

Commits on Apr 8, 2024

  1. Remove startTaskResourceFile object from test 

    Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
    @adamrtalbot
    adamrtalbot committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    c145e45 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into azure_batch_configurable_start_task

    @adamrtalbot
    adamrtalbot authored Apr 8, 2024
    Configuration menu
    Copy the full SHA
    e5c29cf View commit details
    Browse the repository at this point in the history

Commits on Apr 9, 2024

  1. Added ability to run Azure Batch node startTask as privileged (root) … 

    …user.

Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
    @adamrtalbot
    adamrtalbot committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    128e612 View commit details
    Browse the repository at this point in the history

  2. Escalate startTaskPrivileged to pool admin instead of task admin 

    Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
    @adamrtalbot
    adamrtalbot committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    98ce78d View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2024

  1. Add startTask config item to Azure 

    Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
    @adamrtalbot
    adamrtalbot committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    5497860 View commit details
    Browse the repository at this point in the history

  2. Some changes but still not working 

    Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
    @adamrtalbot
    adamrtalbot committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    54d07f4 View commit details
    Browse the repository at this point in the history

  3. Remove Map() 

    Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
    @adamrtalbot
    adamrtalbot committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    c61e10e View commit details
    Browse the repository at this point in the history

  4. Try 'as Map' 

    Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
    @adamrtalbot
    adamrtalbot committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    b53386c View commit details
    Browse the repository at this point in the history

  5. Merge branch 'master' into azure_batch_configurable_start_task

    @adamrtalbot
    adamrtalbot authored Apr 11, 2024
    Configuration menu
    Copy the full SHA
    b5ce455 View commit details
    Browse the repository at this point in the history

  6. tests for new config option 

    Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
    @adamrtalbot
    adamrtalbot committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    21d780c View commit details
    Browse the repository at this point in the history

  7. Docs for Azure startTask options 

    Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
    @adamrtalbot
    adamrtalbot committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    cb24ef2 View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2024

  1. Update plugins/nf-azure/src/main/nextflow/cloud/azure/config/AzPoolOp… 

    …ts.groovy

Co-authored-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
    @adamrtalbot @pditommaso
    adamrtalbot and pditommaso authored Apr 12, 2024
    Configuration menu
    Copy the full SHA
    5b2d406 View commit details
    Browse the repository at this point in the history

  2. Remove getStartTask method 

    Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
    @adamrtalbot
    adamrtalbot committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    c8cc1e5 View commit details
    Browse the repository at this point in the history