Azure Batch configurable startTask

[adamrtalbot]

Azure Batch: Configurable startTask per pool

Additions:

• Make Azure Batch pool startTask configurable.
• Can be used to modifiy the existing worker node. Use cases:
  • Install more recent version of AzCopy
  • Install monitoring software like Azure Insights
  • Use NVME SSDs for storage
  • Flash Dragen firmware to FPGA chips

Pool specific so you can differentiate between different hardware requirements.

Closes #4259

Also spotted a bug with the dev version and the container configuration so fixed it (first commit, can reorganise if we want it in a second PR). Closed with #4854

To use it, you can try this config:

process {
    executor = 'azurebatch'
}

azure {
    storage {
        accountName = "$AZURE_STORAGE_ACCOUNT_NAME"
        accountKey = "$AZURE_STORAGE_ACCOUNT_KEY"
    }
    batch {
        location = "$AZURE_LOCATION"
        accountName = "$AZURE_BATCH_ACCOUNT_NAME"
        accountKey = "$AZURE_BATCH_ACCOUNT_KEY"
        autoPoolMode = true
        deletePoolsOnCompletion = true
        pools {
            auto {
                startTask = "echo 'HELLO WORLD'"
            }
        }
    }
}

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	c8cc1e5
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/661928ab1655b80008d24d02

[adamrtalbot]

I'm not getting these errors locally and they appear to refer to slurping the Google credentials as JSON.

[bentsherman]

I can't reproduce the error locally either. Probably just a flaky google test. I think other PRs have been having a similar issue. The Azure tests are passing which is a good sign, but the integration tests haven't run yet

[pditommaso]

Wonder if the startTask should be added to the default one instead of overriding it

[adamrtalbot]

One of my use cases was to remove the azcopy install. Adding the installer feels unexpected, leaving it out is simple so problem solving should be easier.

And with Fusion you don't need azcopy.

[pditommaso]

I think it makes sense to allow the customisation of the startTask script, however the control over the azcopy should be left to nextflow.

I've added an option to disable explicitly or implicitly when fusion is enabled. See here

#4883

[adamrtalbot]

Seems sensible!

[adamrtalbot]

I'm not 100% clear how the two will interact though. We should probably do something like this:

install_azcopy_cmd = "chmod +x azcopy && mkdir \$AZ_BATCH_NODE_SHARED_DIR/bin/ && cp azcopy \$AZ_BATCH_NODE_SHARED_DIR/bin/"
custom_start_cmd.  = config.startTask ?: ""

final_start_task   = 'bash -c "$custom_start_cmd; $install_azcopy_cmd"'

[pditommaso]

Kind of, but it should be taken in consideration that that now the azcopy command can be absent. See here

nextflow/plugins/nf-azure/src/main/nextflow/cloud/azure/batch/AzBatchService.groovy

Lines 685 to 695 in 74d7d7a

	if( config.batch().getCopyToolInstallMode() != CopyToolInstallMode.node )
	return null
	final resourceFiles = new ArrayList(10)
	resourceFiles << new ResourceFile()
	.withHttpUrl(AZCOPY_URL)
	.withFilePath('azcopy')
	return new StartTask()
	.withCommandLine('bash -c "chmod +x azcopy && mkdir \$AZ_BATCH_NODE_SHARED_DIR/bin/ && cp azcopy \$AZ_BATCH_NODE_SHARED_DIR/bin/" ')
	.withResourceFiles(resourceFiles)

[adamrtalbot]

OK implemented now: 33969d3

[adamrtalbot]

Update: Combine azcopy + custom start task:

process {
    executor = 'azurebatch'
}

azure {
    storage {
        accountName = "$AZURE_STORAGE_ACCOUNT_NAME"
        accountKey = "$AZURE_STORAGE_ACCOUNT_KEY"
    }
    batch {
        location = "$AZURE_LOCATION"
        accountName = "$AZURE_BATCH_ACCOUNT_NAME"
        accountKey = "$AZURE_BATCH_ACCOUNT_KEY"
        autoPoolMode = true
        deletePoolsOnCompletion = true
        pools {
            auto {
                startTask = "echo 'HELLO WORLD3'"
            }
        }
    }
}

Result:

To omit azcopy and use a start task:

process {
    executor = 'azurebatch'
}

azure {
    storage {
        accountName = "$AZURE_STORAGE_ACCOUNT_NAME"
        accountKey = "$AZURE_STORAGE_ACCOUNT_KEY"
    }
    batch {
        location = "$AZURE_LOCATION"
        accountName = "$AZURE_BATCH_ACCOUNT_NAME"
        accountKey = "$AZURE_BATCH_ACCOUNT_KEY"
        autoPoolMode = true
        deletePoolsOnCompletion = true
        pools {
            auto {
                startTask = "echo 'HELLO WORLD2'"
            }
        }
        copyToolInstallMode = 'off'
    }
}

Result:

[pditommaso]

Let's remove the commented code

[adamrtalbot]

Oops. Done now.

[pditommaso]

What if thre's no startTask script and copyToolInstallMode != node? this is going to be empty.

[adamrtalbot]

Ah correct. It should return null instead of an empty task (though I guess an empty task will work first time).

[pditommaso]

It would be nice to add a unit test for createStartTask to check different conditions.

[adamrtalbot]

Now returns null if startTask empty which prevents start task being added:

[adamrtalbot]

Tests added: f9708b0

[adamrtalbot]

Added ability to run the startTask as root user, necessary for modifying the pool hardware.

[pditommaso]

would make sense to run it always as privileged ?

[adamrtalbot]

you don't need to if you want to add some basic software to the PATH (azcopy), so I think configurable makes sense.

[pditommaso]

then, I'd ask to have this organised as

azure.batch.pools.<name>.startTask.script
azure.batch.pools.<name>.startTask.privileged

[adamrtalbot]

I thought you'd say that. I don't know how, give me some time to work it out.

[pditommaso]

it's enough to add a small class

class AzStartTask {
  String script
  boolean unprivilged

  AzStartTask(Map opts) { ... todo }
}

and create it using the usual pattern in the config

[adamrtalbot]

It will be merged to this branch from here, once finished adamrtalbot#1

[adamrtalbot]

All changes in now. startTask is a map:

azure {
    storage {
        accountName = "$AZURE_STORAGE_ACCOUNT_NAME"
        accountKey = "$AZURE_STORAGE_ACCOUNT_KEY"
    }
    batch {
        location = "$AZURE_LOCATION"
        accountName = "$AZURE_BATCH_ACCOUNT_NAME"
        accountKey = "$AZURE_BATCH_ACCOUNT_KEY"
        autoPoolMode = true
        deletePoolsOnCompletion = true
        pools {
            auto {
                startTask {
                    script = "echo 'HELLO WORLD'"
                    privileged = true
                }
            }
        }
    }
}

[pditommaso]

Solved via 27d01e3

[adamrtalbot]

Thanks @pditommaso!