Amazon OpenSearch Ingestion is a strong knowledge ingestion pipeline that AWS prospects use for a lot of totally different functions, reminiscent of observability, analytics, and zero-ETL search. Many shoppers in the present day push logs, traces, and metrics from their functions to OpenSearch Ingestion to retailer and analyze this knowledge.
Right now, we’re pleased to announce that OpenSearch Ingestion pipelines now help cross-account ingestion for push-based sources reminiscent of HTTP and OpenTelemetry (OTel). Organizations can now use this function to effortlessly share knowledge throughout groups. For instance, many organizations have central observability groups—now these groups can create OpenSearch Ingestion pipelines and share them with different groups of their group. You may as well use this function to ingest knowledge into Amazon OpenSearch Service domains or Amazon OpenSearch Serverless collections in different accounts.
Beforehand, sharing OpenSearch Ingestion pipelines throughout accounts required groups to make use of digital personal cloud (VPC) options to share entry. For instance, groups might use VPC peering, which isn’t at all times possible, or AWS Transit Gateway. The brand new cross-account ingestion options in OpenSearch Ingestion can simplify your deployment and cut back value for sharing pipelines.
Resolution overview
Let’s take a look at the right way to share a pipeline from a central logging account with two different growth accounts (A and B). The central logging account can create an OpenSearch Ingestion pipeline utilizing a push-based supply, for instance, HTTP. After creating the pipeline, a member of the central logging workforce can grant entry to the opposite groups. They will use a useful resource coverage that offers permissions to the 2 different workforce accounts to create pipeline endpoints. After making this modification, the OpenSearch Ingestion pipeline is obtainable to be used by the opposite groups.
The next diagram illustrates this configuration.
Within the following sections, we show the right way to implement this resolution.
Conditions
First, the central logging account will need to have a VPC with two choices enabled.
- enableDnsSupport have to be set to true
- enableDnsHostnames have to be set to true
The central logging account should additionally create a push-based OpenSearch Ingestion pipeline within the VPC. This generally is a pipeline receiving logs from FluentBit or OpenTelemetry telemetry.
The event accounts which can be going to hook up with the pipeline additionally will need to have VPCs in the identical area with the identical DNS choices enabled.
- enableDnsSupport have to be set to true
- enableDnsHostnames have to be set to true
Create useful resource coverage
Because the proprietor of the pipeline, you’ll be able to create a useful resource coverage that permits the 2 growth accounts to create pipeline endpoints in opposition to your pipeline.
The next is an instance useful resource coverage for this situation:
The OpenSearch Ingestion console makes it easy to create these insurance policies, as proven within the following screenshot.
Create pipeline endpoint
Now that the central logging account has shared permissions on their pipeline, the event accounts can create pipeline endpoints. A pipeline endpoint is a connection from one VPC to an OpenSearch Ingestion pipeline.
The event accounts are answerable for creating the pipeline endpoints within the VPCs they wish to join from. They create this within the subnets they want and supply a safety group. The safety group ought to have an inbound rule permitting entry port HTTPS over port 443 from any supply that the event accounts have to ingest logs.
Growth workforce A can create a pipeline endpoint utilizing a command just like the next:
Growth workforce A may also use the OpenSearch Ingestion console to create the pipeline endpoint.
After performing this modification, the VPC for growth workforce A may have a pipeline endpoint. This pipeline endpoint now permits for ingesting knowledge into the central logging pipeline. Now, Amazon Elastic Compute Cloud (Amazon EC2) situations, Amazon Elastic Container Service (Amazon ECS) duties, Kubernetes pods, and different compute working within the VPC can ingest their log knowledge into the pipeline utilizing instruments reminiscent of FluentBit.
On the similar time or at a later time, growth workforce B can create a pipeline endpoint as properly. This workforce will create it for their very own VPC.
After this, the pipeline will now have two pipeline endpoints, so each groups can ingest their log knowledge into the central logging VPC.
Clear up
After a pipeline endpoint is created, both account can take away it. The event groups in our situation can use the DeletePipelineEndpoint API to delete it from their accounts. Moreover, if the central logging account must take away a pipeline endpoint from a pipeline, it will probably use the RevokePipelineEndpointConnections API. Each choices can be found on the OpenSearch Ingestion console as properly.
After the pipeline endpoints are eliminated, the central logging workforce may also take away the pipeline in the event that they not want it.
Conclusion
The brand new pipeline endpoint function for OpenSearch Ingestion simplifies how one can share pipelines for cross-account ingestion. This can assist groups use the highly effective options of OpenSearch Ingestion and open up new potentialities for groups or organizations utilizing a number of accounts and VPCs. The brand new pipeline endpoint function is obtainable in the present day in AWS Areas the place OpenSearch Ingestion is obtainable.
To get began with cross-account ingestion in OpenSearch Ingestion, seek advice from OpenSearch Ingestion documentation or attempt creating your first cross-account pipeline on the OpenSearch Ingestion console.
Concerning the authors
