The why, what, and how of MeltanoLabs
We want MeltanoLabs to be a resource for the Singer Community where the expectations of taps and targets are well defined and maintained for the long term. The Singer community has a vast library of connectors but the quality, feature completeness, and maintenance vary. MeltanoLabs is a place where new connectors, primarily built using the Meltano SDK, can be incubated and existing unmaintained connectors in the ecosystem can be migrated and upgraded to the standard that the MeltanoLabs community expects. As part of the acceptance process for MeltanoLabs we will be requiring a level of commitment by the contributors and a minimum level of quality. Based on the Maintenance Statuses we expect that most of the connectors in MeltanoLabs are in the Active (Stable) category and there's a smaller subset of connectors in the Development or Prerelease status because they were either just created or recently migrated.
With MeltanoLabs we hope to foster a community of contributors who are well versed in Singer and the Meltano SDK. The three roles that exist in MeltanoLabs are maintainers, contributors, and community members. Each one has different expectations and level of access to the repositories within MeltanoLabs.
Everyone who uses or contributes to connectors within MeltanoLabs are considered community members. Community members are encouraged to use the connectors available in MeltanoLabs and to submit feedback or raise bugs as issues in the repositories. The more community members we have that are using these connectors the faster we can spot and address bugs while also adding features that are most useful, ultimately making the connectors the best they can be.
MeltanoLabs contributors are community members who volunteer to actively make contributions and review pull requests/issues within MeltanoLabs repositories but don't have an explicit maintainer status on any one repository.
This group of community members will be part of the labs slack channel and will be the first to hear about repositories that need new maintainer volunteers and general maintenance requests.
Connector maintainers are community members who have stepped up to take temporary ownership of a repository's maintenance. Community members can volunteer to be owners of a connector for a specific duration of time. This is much more light weight and very different from asking for full-time permanent fork owners. The developer can support for the project scope or time duration that makes sense. We would likely ask for a 3 months minimum commitment, and/or a commitment to meet a specific core set of functionality - such as reaching 'v1' if the connector is brand new. The expectation is that while a maintainer has long term ownership of a repository they will promptly review issues and pull requests. When the maintainer reaches a point where they no longer have the capacity or desire to be the maintainer anymore they can pass ownership off to another community member volunteer.
MeltanoLabs is introducing the "Community-Managed Fork with Community Maintainers" ownership model by allowing community members to share ownership of connector repositories. The two main ways that connectors are added to MeltanoLabs right now are either by starting fresh and adding a new connector from scratch or by migrating an existing connector from a personal namespace into the shared MeltanoLabs namespace.
MeltanoLabs is a great place for community members to incubate connectors for new sources or destinations in a shared space to make it easier for the community to contribute and maintain it into the future after it leaves the development phase. We also encourage community members to use this as a place where existing connectors can be rebuilt or ported over to the Meltano SDK.
See the Connector Development Standards guide for best practices and conventions.
If you'd like to create a new connector in MeltanoLabs, create an issue labeled "Add New Connector - [your connector name]" in the Meta repo and assign it to @tayloramurphy and @pnadolny13 when you're code is ready to be pushed up to GitHub and they will create you a repository and give you access. Alternatively you could build it in your own namespace and later migrate it once its ready.
Its understandable that sometimes community members build new connector or create a fork and add features in their own namespace which then becomes the primary variant used by the community, but they no longer have the capacity or desire to maintain it. In this case the primary maintainer has the option to migrate their connector to MeltanoLabs where it can live and be maintained into the future.
Connectors in MeltanoLabs have quality expectations. If the connector does not meet the standard then we will label it with the appropriate maintenance status in its current state and request community contributions to get it up to the Active (Stable) status we expect.
If you'd like to migrate an existing connector to MeltanoLabs create an issue labeled "Migrate Existing Connector - [your connector name]" in the Meta repo and assign it to @tayloramurphy and @pnadolny13 with a link to the source repository you would like to migrate. The Admins will review your repository and will give instructions on how to migrate along with necessary permissions once its been approved. Alternatively you could build it in your own namespace and later migrate it once its ready.
Each repository has a GitHub team with the same name e.g. tap-x where community members can be added. The GitHub team is a child of the generic Singer Taps/Targets team. The team is then assigned to the repository with "maintain" permissions. If a maintainer chooses to pass their maintenance responsibilities to another person, they are simply removed from the GitHub team to revoke privileges.
The initial approach will be to ask for volunteers within the MeltanoLabs community to take ownership of a repository as maintainer but we know that for some less frequently used connectors there might not be enough volunteers.
In this case we will do the following:
- Whenever we receive a PR or Issue for a connector that has no maintainers, we announce this to the contributor.
- We will post the PR or Issue to the MeltanoLabs contributors via the
labsSlack channel to request community reviews. Known past contributors and maintainers will get mentioned. - If no one from the community responds in a timely manner, we can offer the PR author to become a contributor if they have a history of contributing to Meltano and/or other open source projects. (We can use their GitHub history to help in this evaluation process.)
- Any successfully-merged PR automatically promotes that person to "contributor" and optionally to maintainer if they've consented in step 3 and no other maintainers exist.
One of the main reasons for the MeltanoLabs ownership model is to allow connector maintainers to come and go. If you're a maintainer and no longer have the capacity or desire to be the maintainer anymore you can pass ownership off to another community member volunteer.
- Open an issue in this repository "Relinquish Maintainership - tap-x" and assign it to @tayloramurphy, @pnadolny13, and @afolson.
- Tag anyone who you suggest to be the next maintainer to get confirmation. It's fine if you don't have someone in mind, we will help find someone from the community.
- Create a PR to edit the connector owner table in this README to remove your name and optionally add the new maintainer if one exists.
- A member of the Meltano team will remove you from the connector's GitHub team and add the new maintainer.
You can request that your package be published to PyPI by opening an issue in this repository "Publish to PyPI - tap-x" and cc'ing @MeltanoLabs/meltanolabs-admin with the following information:
- Repository name, e.g.
MeltanoLabs/tap-messagebird - Desired package name in PyPI
- Release GitHub workflow name, e.g.
release.ymlif the workflow is in.github/workflows/release.yml
After a short review, the team will enable the PyPI publishing workflow for your repository and you'll be able to publish releases to PyPI by triggering the release workflow.
Active (Stable): The connector repository is well maintained and recommended for use in production environments. The repository has all of the following attributes; all known issues and bugs are addressed promptly, changes are expected to go through pull request reviews, releases to main are created on the appropriate cadence, security patches are addressed promptly, a CI pipeline that runs automated testing is present, community contributed pull requests are reviewed in a timely manner, README is complete with examples and detail, etc.
Prerelease (Beta): The repository was recently created and initial development is complete but hasn't been hardened enough for production environments yet. We encourage the community to start testing out the connector and submit issues or merge requests to help get it to the Active (Stable) status. The repository has some or all of the following attributes; most of the features are functioning but edge cases and bugs are still being addressed, changes are expected to go through pull requests and issues, some automated tests but code coverage could be low, optional CI, the README is clear and minimally complete but might be lacking in extensive detail or examples, etc.
Development (Under Construction): The repository is undergoing initial development and is not ready to be used yet. Not recommended for use outside of development environments. The repository has some or all of the following attributes; the developer might be regularly committing directly to the main branch, pull requests reviews are optional, no CI/CD, tests might not be passing, README might be sparse, etc.
Inactive or Stale: The repository has some or all of the following attributes; unresponsive maintainers, many unaddressed issues or pull requests, security patches are not kept up to date, unresolved failing CI pipelines, etc.
Unknown: The status of the connector is currently undefined. We encourage community members to submit merge requests to update incorrect statuses.
See the Connector Development Standards guide for best practices and conventions.
Check out all the published connectors on MeltanoHub!
| Connector Name | Status | Member/Organization Name | MeltanoHub |
|---|---|---|---|
| tap-linkedin-ads | Active (Stable) | Meltano - Maintainer | Link |
| tap-facebook | Active (Stable) | Meltano - Maintainer | Link |
| tap-klaviyo | Active (Stable) | Meltano - Maintainer | Link |
| tap-kustomer | Active (Stable) | Meltano - Maintainer | Link |
| tap-pulumi-cloud | Active (Stable) | Meltano - Maintainer | Link |
| tap-athena | Active (Stable) | Pat Nadolny - Maintainer | Link |
| tap-circle-ci | Active (Stable) | Fran Lozano - Maintainer | Link |
| tap-csv | Active (Stable) | Pat Nadolny - Maintainer | Link |
| tap-dbt | Active (Stable) | Edgar Ramírez - Maintainer | Link |
| tap-github | Active (Stable) | Eric Boucher - Maintainer | Link |
| Laurent Savaete - Maintainer | |||
| tap-gitlab | Active (Stable) | Meltano - Maintainer | Link |
| tap-google-analytics | Active (Stable) | Pat Nadolny - Maintainer | Link |
| tap-peloton | Active (Stable) | Taylor Murphy - Maintainer | Link |
| tap-salesforce | Active (Stable) | Dan Ladd - Maintainer | Link |
| tap-slack | Active (Stable) | Stephen Bailey - Maintainer | Link |
| tap-stackexchange | Active (Stable) | Edgar Ramírez - Maintainer | Link |
| tap-sqlalchemy | Prerelease (Beta) | AJ Steers - Maintainer | |
| tap-sftp | Active (Stable) | Henning - Maintainer | Link |
| tap-cloudwatch | Active (Stable) | Pat Nadolny - Maintainer | Link |
| tap-snowflake | Active (Stable) | Meltano - Maintainer | Link |
| tap-postgres | Active (Stable) | Meltano - Maintainer | Link |
| tap-airbyte-wrapper | Active (Stable) | Alex Butler - Maintainer | Link |
| tap-messagebird | Active (Stable) | Meltano - Maintainer | Link |
| tap-duckdb | Active (Stable) | Meltano - Maintainer | Link |
| Connector Name | Status | Member/Organization Name | MeltanoHub |
|---|---|---|---|
| target-snowflake | Active (Stable) | Meltano - Maintainer | Link |
| target-athena | Active (Stable) | Andrew Stewart - Maintainer | Link |
| AJ Steers - Maintainer | |||
| target-csv | Active (Stable) | AJ Steers - Maintainer | Link |
| target-sqlite | Active (Stable) | Meltano - Maintainer | Link |
| target-postgres | Active (Stable) | Meltano - Maintainer | Link |
| target-yaml | Active (Stable) | Meltano - Maintainer | Link |
| target-jsonl-blob | Prerelease (Beta) | Edgar Ramírez - Maintainer | |
| target-chromadb | Prerelease (Beta) | Meltano - Maintainer | Link |
| target-duckdb | Prerelease (Beta) | Meltano - Maintainer |