Development environment guide #5486
Conversation
Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
✅ Deploy Preview for nextflow-docs-staging ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
|
||
| # Environment setup | ||
|
|
||
| Setting up a Nextflow development environment is a prerequisite for creating, testing, and optimizing data analysis pipelines. The steps below outline recommended tools for setting up an optimal Nextflow development environment. |
There was a problem hiding this comment.
A brief TOC here would help I think, not sure if that's non-standard for these docs though.
docs/developer-env.md
Outdated
| ### Development Containers extension | ||
|
|
||
| The VS Code Dev Containers extension lets you use a container as a full-featured development environment. It allows you to open any folder inside (or mounted into) a container and take advantage of VS Code's full feature set. | ||
|
|
||
| The Dev Containers extension is included as a part of the [Remote Development extension pack](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack). See {ref}`remote-development-ext` for installation instructions. | ||
|
|
||
| ### Create and run a dev container | ||
|
|
||
| A `devcontainer.json` file in your project directory instructs VS Code how to access, create, and configure a development container. It can be used to run an application or to separate tools, libraries, or runtimes. | ||
|
|
||
| The `devcontainer.json` can be used to: | ||
|
|
||
| - Install additional tools in the container. | ||
| - Automatically install extensions. | ||
| - Forward or publish additional ports. | ||
| - Set runtime arguments. | ||
| - Reuse or extend your existing Docker Compose setup. | ||
| - Add more Advanced container configuration. | ||
|
|
||
| To create a dev container with an existing image: | ||
|
|
||
| 1. Create `.devcontainer/devcontainer.json` in the root of your project. | ||
| 1. Add an image with the Nextflow tooling and VS Code customizations to the `devcontainer.json`. For example: | ||
|
|
||
| ```json | ||
| { | ||
| "name": "Nextflow Dev Container", | ||
| "image": "nfcore/gitpod:latest", | ||
| "remoteUser": "vscode", | ||
| "runArgs": ["--privileged"], | ||
|
|
||
| // Configure tool-specific properties. | ||
| "customizations": { | ||
| // Configure properties specific to VS Code. | ||
| "vscode": { | ||
| // Set *default* container specific settings.json values on container create. | ||
| "settings": { | ||
| "python.defaultInterpreterPath": "/opt/conda/bin/python" | ||
| }, | ||
|
|
||
| // Add the IDs of extensions you want installed when the container is created. | ||
| "extensions": ["ms-python.python", "ms-python.vscode-pylance", "nf-core.nf-core-extensionpack", "nextflow.nextflow"] | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| :::{note} | ||
| Instead of using a prebuilt image, a custom Dockerfile may also live in the `.devcontainer` folder. You can replace the image property in `devcontainer.json` with dockerfile and utilize the custom container. See [Create a Dev Container](https://code.visualstudio.com/docs/devcontainers/create-dev-container) for more information. | ||
| ::: | ||
|
|
||
| 1. Enter **Dev Containers: Reopen in Container** in the VS Code Command Palette and reopen your project. You should now see the name of the container ("Nextflow Dev Container" if using the above example) in the bottom left corner of VS Code. | ||
|
|
||
| :::{note} | ||
| Dev Containers can also be used by GitHub Codespaces in VS Code or the browser. See [GitHub Codespaces](https://code.visualstudio.com/docs/remote/codespaces) for more information. |
There was a problem hiding this comment.
not sure this makes much sense in the context of nextflow workflow in which each process can have its own container. Our best practice is Conda + Wave for container reproducibility and portability
There was a problem hiding this comment.
Fair point. I'm not committed to keeping this. It's very thin on details and can get complicated quickly. I'll comment it out.
There was a problem hiding this comment.
I want to push back on p's argument. The intent of a devcontainer isn't to have a single container with all processes for your pipeline (I agree conda + wave is best-practice for that). Instead the devcontainer has all of the pre-requisites like conda, nextflow, nf-core, nf-test, etc installed and updated to their latest versions.
Devcontainers works really well for the following use cases:
- People who are new to developing in Nextflow and don't have any of these tools installed on their computer
- Folks who only occasionally develop Nextflow pipelines and don't want all of those tools cluttering up their system when they're not working on nextflow stuff.
- Teams who want to ensure everyone is working off the same version of all the base tools.
Especially with devcontainers being supported by both gitpod and Codespaces there's also major benefits to being able to boot up into a repo-specific cloud environment quite easily. (part of why I'd love Data Studios to support devcontainers.json down the line too).
There was a problem hiding this comment.
Yeah agree with this. I think that devcontainers for the pipeline development environment is not incompatible with using conda / wave etc for actually running the pipeline.
Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
|
Not sure this is material for nextflow documentation, this is more material for tutorial or blog post |
I've had four people in the last week (1 at summit and 3 at ASHG) tell me that they had a hard time figuring out how to install Nextflow from the documentation. Two of them gave up and quit and never used Nextflow, one of them needed personal help from me, and one was running an ancient version of nextflow that came pre-installed as a workaround. Those user experiences make me think this is a very important addition to the main docs. I also think it would be better featured prominently as one of the first pages new users encounter on the docs page. A separate tutorial page or blog post will be harder to find and won't get continued ongoing attention to make sure we keep it up to date. |
Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Understand and it makes sense. However, I'm contrary having dev containers in this context. Provided it could help, it should not make part of the best practices we are promoting for container usage both for dev and production deployment |
|
|
||
| ### Development Containers extension | ||
|
|
||
| The VS Code Dev Containers extension lets you use a container as a full-featured development environment. It allows you to open any folder inside (or mounted into) a container and take advantage of VS Code's full feature set. |
There was a problem hiding this comment.
We've said "full-featured" 3 times in the past few paragraphs. Maybe we can remove these two or use a different adjective? I'm coming up blank right now though!
Co-authored-by: Justine Geffen <justinegeffen@users.noreply.github.com> Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Co-authored-by: Justine Geffen <justinegeffen@users.noreply.github.com> Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Co-authored-by: Justine Geffen <justinegeffen@users.noreply.github.com> Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Co-authored-by: Justine Geffen <justinegeffen@users.noreply.github.com> Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Co-authored-by: Justine Geffen <justinegeffen@users.noreply.github.com> Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Co-authored-by: Justine Geffen <justinegeffen@users.noreply.github.com> Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Co-authored-by: Justine Geffen <justinegeffen@users.noreply.github.com> Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Co-authored-by: Justine Geffen <justinegeffen@users.noreply.github.com> Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Co-authored-by: Justine Geffen <justinegeffen@users.noreply.github.com> Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Co-authored-by: Justine Geffen <justinegeffen@users.noreply.github.com> Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Co-authored-by: Justine Geffen <justinegeffen@users.noreply.github.com> Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Co-authored-by: Justine Geffen <justinegeffen@users.noreply.github.com> Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Co-authored-by: Justine Geffen <justinegeffen@users.noreply.github.com> Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Co-authored-by: Justine Geffen <justinegeffen@users.noreply.github.com> Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Co-authored-by: Justine Geffen <justinegeffen@users.noreply.github.com> Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Co-authored-by: Justine Geffen <justinegeffen@users.noreply.github.com> Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Co-authored-by: Justine Geffen <justinegeffen@users.noreply.github.com> Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Co-authored-by: Justine Geffen <justinegeffen@users.noreply.github.com> Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Christopher Hakkaart <chris.hakkaart@seqera.io>
bentsherman
added
dependencies
| - [Apptainer/Singularity](https://marketplace.visualstudio.com/items?itemName=onnovalkering.vscode-singularity) - Provides syntax highlighting for Apptainer/Singularity definition files. | ||
| - [Code Spell Checker](https://marketplace.visualstudio.com/items?itemName=streetsidesoftware.code-spell-checker) - A basic spell checker that works well with camelCase code. | ||
| - [Docker](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker) - Makes it easy to create, manage, and debug containerized applications. | ||
| - [EditorConfig](https://marketplace.visualstudio.com/items?itemName=EditorConfig.EditorConfig) - Support for EditorConfig project files for code standardisation. | ||
| - [indent-rainbow](https://marketplace.visualstudio.com/items?itemName=oderwat.indent-rainbow) - Highlight indentation levels, especially if they're inconsistent. | ||
| - [Nextflow](https://marketplace.visualstudio.com/items?itemName=nextflow.nextflow) - Nextflow language support. | ||
| - [Prettier](https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode) - Code formatter using Prettier. | ||
| - [Rainbow CSV](https://marketplace.visualstudio.com/items?itemName=mechatroner.rainbow-csv) - Highlight columns in .CSV files in different colors. | ||
| - [Ruff](https://marketplace.visualstudio.com/items?itemName=charliermarsh.ruff) - An extremely fast Python linter and code formatter, written in Rust. | ||
| - [Todo Tree](https://marketplace.visualstudio.com/items?itemName=Gruntfuggly.todo-tree) - Show TODO, FIXME, etc. comment tags in a tree view. | ||
| - [YAML](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml) - YAML Language Support by Red Hat, with built-in Kubernetes syntax support. | ||
| - [Markdown Extended](https://marketplace.visualstudio.com/items?itemName=jebbs.markdown-extended) - Gives Markdown previews, including admonitions. |
There was a problem hiding this comment.
I'm slightly wary of actually listing them out like this, as it's the kind of thing that could get stale quite quickly. Would prefer something more generic like "a number of useful extensions that come recommended from community members, such as ..." then like 2 or 3 specific examples.
There was a problem hiding this comment.
I can see your point. I'll shrink this (and the other extensions) down
|
|
||
| See {ref}`vs-code-page` for more information about the Nextflow extension. | ||
|
|
||
| ````{tabs} |
There was a problem hiding this comment.
Can we have just one of these for "installing VSCode extensions"? Feels bulky to have essentially the identical content repeated for each set of VSCode extensions.
At the moment the VSCode section is like, most of the page. I think making each subsection shorter would help.
| - [Dev Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) - Work with a separate toolchain or container based application by opening any folder mounted into or inside a container. | ||
| - [WSL](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-wsl) - Get a Linux-powered development experience from the comfort of Windows by opening any folder in the Windows Subsystem for Linux. |
There was a problem hiding this comment.
I'm a bit confused by these as they're not really remote..? Maybe worth addressing this given the heading?
|
|
||
| (wsl)= | ||
|
|
||
| ## Windows Subsystem for Linux |
There was a problem hiding this comment.
This still feels to me to be more part of the Nextflow installation, rather than Dev Environment.
Personally I'd rather remove it from this page entirely.
|
|
||
| See [Set up a WSL development environment](https://learn.microsoft.com/en-us/windows/wsl/setup/environment) for more information. | ||
|
|
||
| ## Development Containers |
There was a problem hiding this comment.
I wonder if this section would be better off as a blog post that we can point to in an admonition from this page.
(a) This part feels quite a lot more detailed than the rest of the page and (b) I feel that devcontainers are more of a "this can be a nice thing" rather than a requirement. I, and I guess the majority, of Nextflow developers have never used this.
There was a problem hiding this comment.
I think it's emerging but not especially common. I'll drop it from this page and add it to the list of possible blog posts (maybe something @kenibrewer would want to write)
There was a problem hiding this comment.
We also have a "Tutorials" section in the Nextflow docs as a temporary home for tutorial-like content until we have something better on seqera.io, so if you want something more permanent than a blog post you can put it there.
Blog post or tutorial would be more appropriate for dev containers for now IMO
There was a problem hiding this comment.
Okay, great. I think it's agreed that it should be removed from the PR. I'll keep the text in case we can recycle it for a blog or tutorial. Depending on how it's pitched, it could be wrapped up into a tutorial or blog. Either way, we can put it aside for now.
Add a new development environment page with the recommended minimal setup. Add tabs to help split Windows/macOS/Linux.
Development environment
@netlify /developer-env