feat: added mirroring support

[Ak-sky]

Description

Added ES mirroring support for disaster recovery usecase.
Git Issue

Release required?

• No release
• Patch release (x.x.X)
• Minor release (x.X.x)
• Major release (X.x.x)

Release notes content

• Added support for mirroring for disaster recovery usecase.

Run the pipeline

If the CI pipeline doesn't run when you create the PR, the PR requires a user with GitHub collaborators access to run the pipeline.

Run the CI pipeline when the PR is ready for review and you expect tests to pass. Add a comment to the PR with the following text:

/run pipeline

Checklist for reviewers

• If relevant, a test for the change is included or updated with this PR.
• If relevant, documentation for the change is included or updated with this PR.

For mergers

• Use a conventional commit message to set the release level. Follow the guidelines.
• Include information that users need to know about the PR in the commit message. The commit message becomes part of the GitHub release notes.
• Use the Squash and merge option.

[JunliWang]

mirroring needs a parameters_json with a structure like this, so that cluster provisioning will come with additional workerpool and mirrormaker application running on it.
source_crn, source_alias, and target_alias are user input and required.

parameters_json = jsonencode(
    {
      mirroring = {
        source_crn   = data.ibm_resource_instance.es_instance_source.id
        source_alias = "source-alias"
        target_alias = "target-alias"
        }
     }
 )

user also has the options to configure it to be more complex like below:

parameters_json = jsonencode(
   {
     mirroring = {
       source_crn   = data.ibm_resource_instance.es_instance_source.id
       source_alias = "source-alias"
       target_alias = "target-alias"
       options = {
         topic_name_transform = {
            type = "rename" 
            rename = {
              add_prefix = "add_prefix"
              add_suffix = "add_suffix"
              remove_prefix = "remove_prefix"
              remove_suffix = "remove_suffix"
            }
         }
         group_id_transform = {
           type = "rename" 
           rename = {
             add_prefix = "add_prefix"
             add_suffix = "add_suffix"
             remove_prefix = "remove_prefix"
             remove_suffix = "remove_suffix"
           }
         }
       }
     }
   }
 )

[kccox]

Note, you need plan = "enterprise-3nodes-2tb" to use mirroring, but provisioning of an enterprise instance can take two to three hours. You probably do not want to test that in pr_test.go.

[Ak-sky]

Addressing changes with the latest stable provider release https://registry.terraform.io/providers/IBM-Cloud/ibm/1.71.0

[JunliWang]

@Ak-sky
the resource ibm_event_streams_mirroring_config depends on the provider release v1.71.0, the parameters_json for mirroring does not, it is already supported by resource controller and Event Streams' service broker.

[JunliWang]

example doc for mirroring
IBM-Cloud/terraform-provider-ibm#5767

[Ak-sky]

@srikant-sahu, can you please have a look at this PR.

[Ak-sky]

/run pipeline

[ocofaigh]

The mirror example will take too long to run in pr_test.go. I think we maybe add a permanent enterprise instance to our account, and update the fscloud example with mirror support. Then we can run the fscloud example in pr_test.go to test all enterprise features

[Ak-sky]

@ocofaigh, should we create this bool variable skip_es_s2s_authorization_policy if the user has this policy already existing?

resource "ibm_iam_authorization_policy" "es_service_policy" {
  count               = var.skip_es_s2s_authorization_policy ? 0 : 1
  source_service_name = "messagehub"
  target_service_name = "messagehub"
  roles               = ["Reader"]
  description         = "Required for provisioning mirroring instance."
}

[ocofaigh]

thats a odd policy - the service needs to grant itself reader access? If anything this policy should probably be scoped to exact instances if possible. And yes always provide the ability to skip s2s auth policy creation

[Ak-sky]

@ocofaigh
So there are 2 policies required for mirroring-
1- service-to-service (The odd one, which is required for ES mirror instance provisioning.)
2- instance-to-instance (This is to allow source cluster access to target cluster.)

I discussed the same with ES team, for which they mentioned both the policies are required for the mirroring feature.

@srikant-sahu/@JunliWang, can you please pitch in here for service-to-service policy need?

[JunliWang]

the expected workflow for mirroring

• provision source cluster
• create s2s policy at service level (at this point we do not have target cluster CRN yet, so grant at service level)
• provision target cluster (this requires s2s policy because mirrormaker2 on target cluster will be configured to read from source cluster, reader role is needed)
• create s2s policy at instance level (in order to ensure target can read from source cluster, at this point both CRNs are available)
• delete s2s policy at service level (can be done via terraform destroy xxx)

[ocofaigh]

OK so it seems we can at least scope the auth policy to the source instance anyway. And we can even scope it to the target resource group since we know what that will be

[ocofaigh]

@JunliWang The approach you have mentioned around deleting the global auth policy and replacing it with a finer scoped one after the target instance is created does not work with terraform (or at least is not something we can automate in our module)

[srikant-sahu]

the variable mirroring needs to have the full mirroring configuration. A user has an option to enable/disbale topic/group renaming. For reference of a full mirroring configuration options see the #327 (comment)