Implement fine-grained entry management for Iceberg tables utilizing Amazon EMR on EKS built-in with AWS Lake Formation

October 26, 2025

61

The rise of distributed information processing frameworks corresponding to Apache Spark has revolutionized the best way organizations handle and analyze large-scale information. Nonetheless, as the quantity and complexity of information proceed to develop, the necessity for fine-grained entry management (FGAC) has turn out to be more and more vital. That is notably true in eventualities the place delicate or proprietary information have to be shared throughout a number of groups or organizations, corresponding to within the case of open information initiatives. Implementing sturdy entry management mechanisms is essential to take care of safe and managed entry to information saved in Open Desk Format (OTF) inside a contemporary information lake.

One method to addressing this problem is by utilizing Amazon EMR on Amazon Elastic Kubernetes Service (Amazon EKS) and incorporating FGAC mechanisms. With Amazon EMR on EKS, you possibly can run open supply huge information frameworks corresponding to Spark on Amazon EKS. This integration gives the scalability and adaptability of Kubernetes, whereas additionally utilizing the information processing capabilities of Amazon EMR.

On February 6^th 2025, AWS launched fine-grained entry management primarily based on AWS Lake Formation for EMR on EKS from Amazon EMR 7.7 and better model. Now you can considerably improve your information governance and safety frameworks utilizing this characteristic.

On this publish, we show the right way to implement FGAC on Apache Iceberg tables utilizing EMR on EKS with Lake Formation.

Information mesh use case

With FGAC in a information mesh structure, area house owners can handle entry to their information merchandise at a granular degree. This decentralized method permits for larger agility and management, ensuring information is accessible solely to licensed customers and providers inside or throughout domains. Insurance policies might be tailor-made to particular information merchandise, contemplating components like information sensitivity, person roles, and supposed use. This localized management enhances safety and compliance whereas supporting the self-service nature of the information mesh.

FGAC is particularly helpful in enterprise domains that cope with delicate information, corresponding to healthcare, finance, authorized, human assets, and others. On this publish, we deal with examples from the healthcare area, showcasing how we are able to obtain the next:

Share affected person information securely – Information mesh permits totally different departments inside a hospital to handle their very own affected person information as unbiased domains. FGAC makes certain solely licensed personnel can entry particular affected person data or information components primarily based on their roles and need-to-know foundation.
Facilitate analysis and collaboration – Researchers can entry de-identified affected person information from varied hospital domains by way of the information mesh structure, enabling collaboration between multidisciplinary groups throughout totally different healthcare establishments, fostering data sharing, and accelerating analysis and discovery. FGAC helps compliance with privateness laws (corresponding to HIPAA) by limiting entry to delicate information components or permitting entry solely to aggregated, anonymized datasets.
Enhance operational effectivity – Information mesh can streamline information sharing between hospitals and insurance coverage firms, simplifying billing and claims processing. FGAC makes certain solely licensed personnel inside every group can entry the required information, defending delicate monetary data.

Resolution overview

On this publish, we discover the right way to implement FGAC on Iceberg tables inside an EMR on EKS utility, utilizing the capabilities of Lake Formation. For particulars on the right way to implement FGAC on Amazon EMR on EC2, discuss with High-quality-grained entry management in Amazon EMR Serverless with AWS Lake Formation.

The next parts play essential roles on this answer design:

Apache Iceberg OTF:
- Excessive-performance desk format for large-scale analytics
- Helps schema evolution, ACID transactions, and time journey
- Suitable with Spark, Trino, Presto, and Flink
- Amazon S3 Tables absolutely managed Iceberg tables for analytics workload
AWS Lake Formation:
- FGAC for information lakes
- Column-, row-, and cell-level safety controls
Information mesh producers and customers:
- Producers: Create and serve domain-specific information merchandise
- Shoppers: Entry and combine information merchandise
- Permits self-service information consumption

To show how you should use Lake Formation to implement cross-account FGAC inside an EMR on EKS surroundings, we create tables within the AWS Glue Information Catalog in a central AWS account appearing as producer and provision totally different person personas to replicate varied roles and entry ranges in a separate AWS account appearing as a number of customers. Shoppers might be unfold throughout a number of accounts in real-world eventualities.

The next diagram illustrates the high-level answer structure.

Determine 1: Excessive Degree Resolution Structure

To show the cross-account information sharing and information filtering with Lake Formation FGAC, the answer deploys two totally different Iceberg tables with assorted entry for various customers. The permission mapping for customers are with cross-account desk shares and information cell filters.

It has two totally different groups with totally different ranges of Lake Formation permissions to entry Sufferers and Claims Iceberg tables. The next desk summarizes the answer’s person personas.

Persona/Desk Title Sufferers Claims

Sufferers Care Staff

(team1 job execution position)

Exclude a column ssn
Embrace rows solely from Texas and New York states

Full desk entry

Claims Care Staff

(team2 job execution position)

No entry

Full desk entry

Stipulations

This answer requires an AWS account with an AWS Id and Entry Administration (IAM) energy person position that may create and work together with AWS providers, together with Amazon EMR, Amazon EKS, AWS Glue, Lake Formation, and Amazon Easy Storage Service (Amazon S3). Further particular necessities for every account are detailed within the related sections.

Clone the mission

To get began, obtain the mission both to your pc or the AWS CloudShell console:

git clone https://github.com/aws-samples/sample-emr-on-eks-fgac-iceberg
 cd sample-emr-on-eks-fgac-iceberg

Arrange infrastructure in producer account

To arrange the infrastructure within the producer account, you could have the next extra assets:

The setup script deploys the next infrastructure:

An S3 bucket to retailer pattern information in Iceberg desk format, registered as a knowledge location in Lake Formation
An AWS Glue database named healthcare_db
Two AWS Glue tables: Sufferers and Claims Iceberg tables
A Lake Formation information entry IAM position
Cross-account permissions enabled for the patron account:
- Permit the patron to explain the database healthcare_db within the producer account
- Permit to entry the Sufferers desk utilizing a knowledge cell filter, primarily based on row-level chosen state, and exclude column ssn
- Permit full desk entry to the Claims desk

Run the next producer_iceberg_datalake_setup.sh script to create a improvement surroundings within the producer account. Replace its parameters in keeping with your necessities:

export AWS_REGION=us-west-2
export PRODUCER_AWS_ACCOUNT= 
export CONSUMER_AWS_ACCOUNT= 
./producer_iceberg_datalake_setup.sh 
# run the clean-up script earlier than re-run the setup if wanted
./producer_clean_up.sh

Allow cross-account Lake Formation entry in producer account

A client account ID and an EMR on EKS Engine session tag should set within the producer’s surroundings. It permits the patron to entry the producer’s AWS Glue tables ruled by Lake Formation. Full the next steps to allow cross-account entry:

Open the Lake Formation console within the producer account.
Select Utility integration settings underneath Administration within the navigation pane.
Choose Permit exterior engines to filter information in Amazon S3 areas registered with Lake Formation.
For Session tag values, enter EMR on EKS Engine.
For AWS account IDs, enter your client account ID.
Select Save.

Comprehensive AWS Lake Formation application integration settings interface for managing third-party data access.

Determine 2: Producer Account – Lake Formation third-party engine configuration display with session tags, account IDs, and information entry permissions.

Validate FGAC setup in producer surroundings

To validate the FGAC setup within the producer account, verify the Iceberg tables, information filter, and FGAC permission settings.

Iceberg tables

Two AWS Glue tables in Iceberg format have been created by producer_iceberg_datalake_setup.sh. On the Lake Formation console, select Tables underneath Information Catalog within the navigation pane to see the tables listed.

AWS Lake Formation Tables interface showing a success message for updated external data filtering settings, with a table list displaying healthcare database tables in Apache Iceberg format.

Determine 3: Lake Formation interface displaying claims and sufferers tables from healthcare_db with Apache Iceberg format.

The next screenshot exhibits an instance of the sufferers desk information.

Determine 4: Sufferers desk information

The next screenshot exhibits an instance of the claims desk information.

Determine 5: Claims desk information

Information cell filter in opposition to sufferers desk

After efficiently working the producer_iceberg_datalake_setup.sh script, a brand new information cell filter named patients_column_row_filter was created in Lake Formation. This filter performs two features:

Exclude the ssn column from the sufferers desk information
Embrace rows the place the state is Texas or New York

To view the information cell filter, select Information filters underneath Information Catalog within the navigation pane of the Lake Formation console, and open the filter. Select View permission to view the permission particulars.

Determine 6: Column and Row degree filter configuration for sufferers desk

FGAC permissions permitting cross-account entry

To view all of the FGAC permissions, select Information permissions underneath Permissions within the navigation pane of the Lake Formation console, and filter by the database identify healthcare_db.

Make certain to revoke information permissions with the IAMAllowedPrincipals principal related to the healthcare_db tables, as a result of it would trigger cross-account information sharing to fail, notably with AWS Useful resource Entry Supervisor (AWS RAM).

Determine 7: Lake Formation information permissions interface displaying filtered healthcare database assets with granular entry controls

The next desk summarizes the general FGAC setup.

Useful resource Sort	Useful resource	Permissions	Grant Permissions
Database		Describe	Describe
Information Cell Filter	`patients_column_row_filter`	Choose	Choose
Desk		Choose, Describe	Choose, Describe

Arrange infrastructure in client account

To arrange the infrastructure within the client account, you could have the next extra assets:

eksctl and kubectl packages have to be put in
An IAM position within the client account have to be a Lake Formation administrator to run consumer_emr_on_eks_setup.sh script
The Lake Formation admin should settle for the AWS RAM useful resource share invitations utilizing the AWS RAM console, if the patron account is outdoors of the producer’s organizational unit

Determine 8: Client account – Cross-account RAM share for Lake Formation useful resource

The setup script deploys the next infrastructure:

An EKS cluster referred to as fgac-blog with two namespaces:
- Person namespace: lf-fgac-user
- System namespace:lf-fgac-secure
An EMR on EKS digital cluster emr-on-eks-fgac-blog:
- Arrange with a safety configuration emr-on-eks-fgac-sec-conifg
- Two EMR on EKS job execution IAM roles:
  - Position for the Sufferers Care Staff (team1): emr_on_eks_fgac_job_team1_execution_role
  - Position for Claims Care Staff (team2): emr_on_eks_fgac_job_team2_execution_role
- A question engine IAM position utilized by FGAC safe house: emr_on_eks_fgac_query_execution_role
An S3 bucket to retailer PySpark job scripts and logs
An AWS Glue native database named consumer_healthcare_db
Two useful resource hyperlinks to cross-account shared AWS Glue tables: rl_patients and rl_claims
Lake Formation permission on Amazon EMR IAM roles

Run the next consumer_emr_on_eks_setup.sh script to arrange a improvement surroundings within the client account. Replace the parameters in keeping with your use case:

export AWS_REGION=us-west-2 
export PRODUCER_AWS_ACCOUNT= 
export EKSCLUSTER_NAME=fgac-blog 
./consumer_emr_on_eks_setup.sh 
# run the clean-up script earlier than re-run the setup if wanted
./consumer_clean_up.sh

Allow cross-account Lake Formation entry in client account

The patron account should add the patron account ID with an EMR on EKS Engine session tag in Lake Formation. This session tag will likely be utilized by EMR on EKS job execution IAM roles to entry Lake Formation tables. Full the next steps:

Open the Lake Formation console within the client account.
Select Utility integration settings underneath Administration within the navigation pane.
Choose Permit exterior engines to filter information in Amazon S3 areas registered with Lake Formation.
For Session tag values, enter EMR on EKS Engine.
For AWS account IDs, enter your client account ID.
Select Save.

Determine 9: Client Account – Lake Formation third-party engine configuration display with session tags, account IDs, and information entry permissions

Validate FGAC setup in client surroundings

To validate the FGAC setup within the producer account, verify the EKS cluster, namespaces, and Spark job scripts to check information permissions.

EKS cluster

On the Amazon EKS console, select Clusters within the navigation pane and make sure the EKS cluster fgac-blog is listed.

Determine 10: Client Account – EKS Cluster console web page

Namespaces in Amazon EKS

Kubernetes makes use of namespaces as logical partitioning system for organizing objects corresponding to Pods and Deployments. Namespaces additionally function as a privilege boundary within the Kubernetes role-based entry management (RBAC) system. Multi-tenant workloads in Amazon EKS might be secured utilizing namespaces.

This answer creates two namespaces:

lf-fgac-user
lf-fgac-secure

The StartJobRun API makes use of the backend workflows to submit a Spark job’s UserComponents (JobRunner, Driver, Executors) within the person namespace, and the corresponding system parts within the system namespace to perform the specified FGAC behaviors.

You possibly can confirm the namespaces with the next command:kubectl get namespaceThe next screenshot exhibits an instance of the anticipated output.

Determine 11: EKS Cluster namespaces

Spark job script to check Sufferers Care Staff’s information permissions

Beginning with Amazon EMR model 6.6.0, you should use Spark on EMR on EKS with the Iceberg desk format. For extra data on how Iceberg works in an immutable information lake, see Construct a high-performance, ACID compliant, evolving information lake utilizing Apache Iceberg on Amazon EMR.

The next script is a snippet of the PySpark job that retrieves filtered information for the Claims and Affected person tables:

    print("Affected person Care Staff PySpark job working on EMR on EKS! to question Sufferers and Claims tables!")
    print("This job queries Sufferers and Claims tables!")
    df1 = spark.sql('SELECT * FROM dev.${CONSUMER_DATABASE}.${rl_patients}')
    print("Sufferers tables information:")
    print("Observe: Sufferers desk is filtered on SSN column and it exhibits data just for Texas and New York states")
    df1.present(20)
    df2 = spark.sql('SELECT p.state,
                            c.claim_id,
                            c.claim_date, 
                            p.patient_name, 
                            c.diagnosis_code, 
                            c.procedure_code, 
                            c.quantity, 
                            c.standing, 
                            c.provider_id 
                    FROM dev.${CONSUMER_DATABASE}.${rl_claims} c 
                    JOIN dev.${CONSUMER_DATABASE}.${rl_patients} p
                   ON c.patient_id = p.patient_id 
                   ORDER BY p.state, c.claim_date')
    print("Present solely related Claims information for Sufferers chosen from Texas and New York state:")
    df2.present(20)
    print("Job Full")
....

Spark job script to check Claims Care Staff’s information permissions

The next script is a snippet of the PySpark job that retrieves information from the Claims desk:

    print("Claims Staff PySpark job working on EMR on EKS to question Claims desk!")
    print("Observe: Claims Staff has full entry to Claims desk!")
    df = spark.sql('SELECT * FROM     dev.${CONSUMER_DATABASE}.${rl_claims}')
    df.present(20)
....

Validate job execution roles for EMR on EKS

The Sufferers Care Staff makes use of the emr_on_eks_fgac_job_team1_execution_role IAM position to execute a PySpark job on EMR on EKS. The job execution position has permission to question each the Sufferers and Claims tables.

The Claims Care Staff makes use of the emr_on_eks_fgac_job_team2_execution_role IAM position to execute jobs on EMR on EKS. The job execution position solely has permission to entry Claims information.

Each IAM job execution roles have the next permissions:

{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Sid": "EmrGetCertificate",
            "Effect": "Allow",
            "Action": "emr-containers:CreateCertificate",
            "Resource": "*"
        },
        {
            "Sid": "LakeFormationManagedAccess",
            "Effect": "Allow",
            "Action": [
                "lakeformation:GetDataAccess",
                "glue:GetTable",
                "glue:GetCatalog",
                "glue:Create*",
                "glue:Update*"
            ],
            "Useful resource": "*"
        },
        {
            "Sid": "EmrSparkJobAccess",
            "Impact": "Permit",
            "Motion": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:ListBucket"
            ],
            "Useful resource": [
                "arn:aws:s3:::${S3_BUCKET}*"
            ]
        }
        }
    ]
}

The next code is the job execution IAM position belief coverage:

{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Sid": "TrustQueryEngineRoleToAssume",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::$CONSUMER_ACCOUNT:role/$query_engine_role"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ],
            "Situation": {
                "StringLike": {
                    "aws:RequestTag/LakeFormationAuthorizedCaller": "EMR on EKS Engine"
                }
            }
        },
        {
            "Sid": "TrustQueryEngineRoleToAssumeRoleOnly",
            "Impact": "Permit",
            "Principal": {
                "AWS": "arn:aws:iam::$CONSUMER_ACCOUNT:position/$query_engine_role"
            },
            "Motion": "sts:AssumeRole"
        },
        {
            "Impact": "Permit",
            "Principal": {
                "Federated": "arn:aws:iam::$CONSUMER_ACCOUNT oidc-provider/oidc.eks.$AWS_REGION.amazonaws.com/id/xxxxx"
            },
            "Motion": "sts:AssumeRoleWithWebIdentity",
            "Situation": {
                "StringLike": {
                    "oidc.eks.$AWS_REGION.amazonaws.com/id/xxxxx:sub": "system:serviceaccount:lf-fgac-user:emr-containers-sa-*-*-$CONSUMER_ACCOUNT-"
                }
            }
        }
    ]
}

The next code is the question engine IAM position coverage (emr_on_eks_fgac_query_execution_role-policy):

{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Sid": "AssumeJobExecutionRole",
            "Effect": "Allow",
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ],
            "Useful resource": ["arn:aws:iam::$CONSUMER_ACCOUNT:role/emr_on_eks_fgac_job_team1_execution_role",
                "arn:aws:iam::$CONSUMER_ACCOUNT:role/emr_on_eks_fgac_job_team2_execution_role"],
            "Situation": {
                "StringLike": {
                    "aws:RequestTag/LakeFormationAuthorizedCaller": "EMR on EKS Engine"
                }
            }
        },
        {
            "Sid": "AssumeJobExecutionRoleOnly",
            "Impact": "Permit",
            "Motion": [
                "sts:AssumeRole"
            ],
            "Useful resource": [
                "arn:aws:iam::$CONSUMER_ACCOUNT:role/emr_on_eks_fgac_job_team1_execution_role",
                "arn:aws:iam::$CONSUMER_ACCOUNT:role/emr_on_eks_fgac_job_team2_execution_role"
            ]
    ]
}

The next code is the question engine IAM position belief coverage:

{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::$CONSUMER_ACCOUNT:root"
            },
            "Action": "sts:AssumeRole",
            "Condition": {}
        },
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::$CONSUMER_ACCOUNT:oidc-provider/xxxxx"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringLike": {
                    "xxxxxx:sub": "system:serviceaccount:lf-fgac-secure:emr-containers-sa-*-*-$CONSUMER_ACCOUNT-"
                }
            }
        }
    ]
}

Run PySpark jobs on EMR on EKS with FGAC

For extra particulars about the right way to work with Iceberg tables in EMR on EKS jobs, discuss with Utilizing Apache Iceberg with Amazon EMR on EKS. Full the next steps to run the PySpark jobs on EMR on EKS with FGAC:

Run the next instructions to run the sufferers and claims jobs:

bash /tmp/submit-patients-job.sh
bash /tmp/submit-claims-job.sh

Watch the appliance logs from the Spark driver pod:

kubectl logs drive-pod-name -c spark-kubernetes-driver -n lf-fgac-user -f

Alternatively, you possibly can navigate to the Amazon EMR console, open your digital cluster, and select the open icon subsequent to the job to open the Spark UI and monitor the job progress.

Determine 12: EMR on EKS job runs

View PySpark jobs output on EMR on EKS with FGAC

In Amazon S3, navigate to the Spark output logs folder:

s3://blog-emr-eks-fgac-test--us-west-2-dev/spark-logs//jobs//containers/spark-xxxxxx/spark-xxxxx-driver/stdout.gz

Determine 13: EMR on EKS job’s stdout.gz location on S3 Bucket

The Sufferers Care Staff PySpark job has question entry to the Sufferers and Claims tables. The Sufferers desk has filtered out the SSN column and solely exhibits data for Texas and New York declare data, as laid out in our FGAC setup.

The next screenshot exhibits the Claims desk for less than Texas and New York.

Determine 14: EMR on EKS Spark job output

The next screenshot exhibits the Sufferers desk with out the SSN column.

Determine 15: EMR on EKS Spark job output

Equally, navigate to the Spark output log folder for the Claims Care Staff job:

s3://blog-emr-eks-fgac-test--us-west-2-dev/spark-logs//jobs//containers/spark-xxxxxx/spark-xxxxx-driver/stdout.gz

As proven within the following screenshot, the Claims Care Staff solely has entry to the Claims desk, so when the job tried to entry the Sufferers desk, it obtained an entry denied error.

Determine 16: EMR on EKS Spark job output

Issues and limitations

Though the method mentioned on this publish gives beneficial insights and sensible implementation methods, it’s vital to acknowledge the important thing concerns and limitations earlier than you begin utilizing this characteristic. To be taught extra about utilizing EMR on EKS with Lake Formation, discuss with How Amazon EMR on EKS works with AWS Lake Formation.

Clear up

To keep away from incurring future expenses, delete the assets generated when you don’t want the answer anymore. Run the next cleanup scripts (change the AWS Area if crucial).Run the next script within the client account:

export AWS_REGION=us-west-2
export PRODUCER_AWS_ACCOUNT=
export EKSCLUSTER_NAME=fgac-blog
./consumer_clean_up.sh

Run the next script within the producer account:

export AWS_REGION=us-west-2
export PRODUCER_AWS_ACCOUNT=
export CONSUMER_AWS_ACCOUNT=
./producer_clean_up.sh

Conclusion

On this publish, we demonstrated the right way to combine Lake Formation with EMR on EKS to implement fine-grained entry management on Iceberg tables. This integration affords organizations a contemporary method to imposing detailed information permissions inside a multi-account open information lake surroundings. By centralizing information administration in a main account and thoroughly regulating person entry in secondary accounts, this technique can simplify governance and improve safety.

For extra details about Amazon EMR 7.7 in reference to EMR on EKS, see Amazon EMR on EKS 7.7.0 releases. To be taught extra about utilizing Lake Formation with EMR on EKS, see Allow Lake Formation with Amazon EMR on EKS.

We encourage you to discover this answer in your particular use instances and share your suggestions and questions within the feedback part.