How to Securely Host Cloud Run for Each Pull Request Using Identity-Aware Proxy

Hello, I’m riddle from the SRE Group at MIXI Development Division.

Previously, I wrote an article about Securely Hosting ‘Flutter on the Web’ and ‘WidgetBook’ Apps on GCS with Individual GitHub Pull Requests.This time I’ll try something similar with Cloud Run.

Hosting Cloud Run for each Pull Request is simple, but using Identity-Aware Proxy can be tricky, so I’ll focus on that.

Table of Contents:

1. Structure Creation Overview
2. How It Works
   2.1 Developer Accesses https://XXXXX.example.com
   2.2 Identity-Aware Proxy Authenticates the Developer
   2.3 Nginx Reverse Proxy to Cloud Run
   2.4 Request to Cloud Run
3. Cloud Run YAML File
4. Conclusion
5. References

Structure Creation Overview

The flow when creating a Pull Request (PR) is as follows:

1. A PR is created on GitHub.
2. GitHub Actions deploy Cloud Run, obtaining the URL https://XXXXX.a.run.app from Cloud Run.
3. GitHub Actions add an A record for XXXXX.example.com using the IP address used by the LoadBalancer (LB).

Access works as follows:

1. Developer accesses https://XXXXX.example.com.
2. Identity-Aware Proxy authenticates/authorizes the Developer.
3. If authentication/authorization is successful, access is granted.

How It Works

I’ll explain the communication flowchart below.

1. Developer Accesses https://XXXXX.example.com

Since GitHub Actions have registered an A record, when the Developer accesses https://XXXXX.example.com, it reaches the LB. SSL Termination is performed at the LB for HTTPS communication.

Google Cloud’s Certificate Manager can issue wildcard certificates, so we use it to issue certificates for *.example.com. This allows certificate verification for any subdomain (e.g., https://hogehoge.example.com).

2. Identity-Aware Proxy Authenticates the Developer

Using the Identity-Aware Proxy, we can authenticate/authorize the Developer. Therefore, we grant httpsResourceAccessor permission to the Google account of the Developer we want to allow access.

3. Nginx Reverse Proxy to Cloud Run

Once Authentication/Authorization is successful through the Identity-Aware Proxy, Nginx is used to reverse proxy to Cloud Run. Here, Nginx reverse proxies to https://XXXXX.run.app. This part is fairly complicated.

Here is the kind of config I wrote:

# default Cloud Run Resolver is 169.254.1.1
# but this resolver does not work on Cloud Run to resolve subdomain.a.run.app
resolver 8.8.8.8;

server {
   listen 80;
   server_name  ~^(?.+)\.example\.com$;
   location / {

       # set Authorization header by /token/auth_token
       rewrite_by_lua_block {
           local file = io.open("/token/auth_token")
           io.input(file)
           local data = io.read()
           io.close()
           ngx.req.set_header("Authorization", "Bearer "..data);

           local subdomain = ngx.var.subdomain
       }

       # forwarding
       proxy_pass https://$subdomain.a.run.app;
       proxy_set_header Host $subdomain.a.run.app;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;

       # remove X-Serverless-Authorization header
       # IAP add requester account's OIDC ID Token to this header
       # and this header use by Google Front End instead of Authorization header
       proxy_set_header X-Serverless-Authorization "";
   }
}

Some points to note:

The default DNS resolver set in Cloud Run cannot resolve subdomain.a.run.app, so I point the resolver externally.

# default Cloud Run Resolver is 169.254.1.1
# but this resolver does not work on Cloud Run to resolve subdomain.a.run.app
resolver 8.8.8.8;

When accessing Cloud Run, it’s necessary to attach the ID Token of an account with Cloud Run Invoker to the Authorization header. Therefore, I write the ID Token to /token/auth_token and set it in the Nginx Authorization header.

※Inspired by: https://zenn.dev/shin5ok/articles/abe99ad833aeee#nginx%E3%81%AE%E8%A8%AD%E5%AE%9A

# set Authorization header by /token/auth_token
rewrite_by_lua_block {
   local file = io.open("/token/auth_token")
   io.input(file)
   local data = io.read()
   io.close()
   ngx.req.set_header("Authorization", "Bearer "..data);

   local subdomain = ngx.var.subdomain
}

The X-Serverless-Authorization header is added by IAP when sending a request to Cloud Run (Authenticating service-to-service | Google Cloud).

This header is prioritized by Google Front End over the Authorization header, so I remove it.

# remove X-Serverless-Authorization header
# IAP add requester account's OIDC ID Token to this header
# and this header use by Google Front End instead of Authorization header
proxy_set_header X-Serverless-Authorization "";

4. Request to Cloud Run

Cloud Run, which is publicly accessible and requires authentication, is normally inaccessible, but can be accessed by Nginx attaching the Authorization header to the request.

This allows Developers to access Cloud Run.

Cloud Run YAML File

Here is the actual Cloud Run YAML file I used:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
 annotations:
   run.googleapis.com/ingress: internal-and-cloud-load-balancing
   run.googleapis.com/launch-stage: BETA
 labels:
   cloud.googleapis.com/location: asia-northeast1
 name: pr-cloudrun-with-iap
spec:
 template:
   metadata:
     annotations:
       autoscaling.knative.dev/maxScale: "1"
       run.googleapis.com/container-dependencies: '{"proxy":["config-generator", "token-generator"]}'
   spec:
     # add Cloud Run Invoker Role / Storage Object Viewer Role
     serviceAccountName: pr-cloudrun@[YOUR_PROJECT].iam.gserviceaccount.com
     containerConcurrency: 80
     containers:
       - name: proxy
         image: openresty/openresty:latest
         ports:
           - containerPort: 80
             name: http1
         resources:
           limits:
             cpu: 1000m
             memory: 512Mi
         startupProbe:
           tcpSocket:
             port: 80

         volumeMounts:
           - name: token
             readOnly: true
             mountPath: /token
           - name: config
             readOnly: true
             mountPath: /etc/nginx/conf.d

       - name: config-generator
         image: gcr.io/google.com/cloudsdktool/google-cloud-cli:slim
         command: ["/bin/sh"]
         args:
           - "-c"
           - |

             cat << "EOF" > /config/proxy.conf

             # default Cloud Run Resolver is 169.254.1.1
             # should be changed to resolve subdomain.a.run.app
             resolver 8.8.8.8;

             server {
                 listen 80;
                 server_name  ~^(?.+)\.example\.com$;
                 location / {

                     # set Authorization header by /token/auth_token
                     rewrite_by_lua_block {
                         local file = io.open("/token/auth_token")
                         io.input(file)
                         local data = io.read()
                         io.close()
                         ngx.req.set_header("Authorization", "Bearer "..data);

                         local subdomain = ngx.var.subdomain
                     }

                     # forwarding
                     proxy_pass https://$subdomain.a.run.app;
                     proxy_set_header Host $subdomain.a.run.app;
                     proxy_set_header X-Real-IP $remote_addr;
                     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                     proxy_set_header X-Forwarded-Proto $scheme;

                     # remove X-Serverless-Authorization header
                     # IAP add requester account's OIDC ID Token to this header
                     # and this header use by Google Front End instead of Authorization header
                     proxy_set_header X-Serverless-Authorization "";
                 }
             }
             EOF

             # Start a simple HTTP server for keep container
             mkdir -p /server
             cd /server
             echo "I'm alive" > index.html
             python3 -m http.server 8081

         # / is redirect to other cloud run service
         # so, use tcp probe instead of http probe
         startupProbe:
           tcpSocket:
             port: 8081

         volumeMounts:
           - name: config
             readOnly: true
             mountPath: /config

       - name: token-generator
         image: gcr.io/google.com/cloudsdktool/google-cloud-cli:slim
         command: ["/bin/sh"]
         args:
           - "-c"
           - |
             # create first, and avoid no file error
             gcloud auth print-identity-token > /token/auth_token

             # Start a simple HTTP server for keep container
             mkdir -p /server
             cd /server
             echo "I'm alive" > index.html
             python3 -m http.server 8082

             # update token every 3500 seconds
             # because token is expired after 3600 seconds
             while true; do gcloud auth print-identity-token > /token/auth_token ; sleep 3500; done

         startupProbe:
           tcpSocket:
             port: 8082

         volumeMounts:
           - name: token
             readOnly: true
             mountPath: /token

     timeoutSeconds: 300
     volumes:
       - name: token
         emptyDir:
           medium: Memory
       - name: config
         emptyDir:
           medium: Memory
 traffic:
   - latestRevision: true
     percent: 100

Some points to note:

The Service Account assigned to Cloud Run has the Cloud Run Invoker Role and Storage Object Viewer Role. The Cloud Run Invoker Role is for accessing other Cloud Run services.

I also assigned the Storage Object Viewer Role because it was necessary to obtain the official image from Google Container Registry.

serviceAccountName: pr-cloudrun@[YOUR_PROJECT].iam.gserviceaccount.com

This YAML starts three containers, including two sidecars:

1. proxy: Nginx reverse proxy.
2. config-generator: Container to generate Nginx config.
3. token-generator: Container to generate ID Token for Cloud Run Invoker.

The proxy, as explained earlier, uses the OpenResty image, which includes various 3rd party modules for Lua.

The config-generator is a container for generating Nginx config, as ConfigMap is not available in Cloud Run.

The token-generator generates the ID Token for Cloud Run connection. The ID Token attached to the Authorization header earlier needs to have the Cloud Run Invoker Role, so it’s generated using the permissions of the ServiceAccount attached to Cloud Run itself.

The generated ID Token expires after 1 hour, so it’s generated at container startup and then updated with time to spare.

while true; do gcloud auth print-identity-token > /token/auth_token ; sleep 3500; done

Config and ID Token are shared between containers using Volumes, allowing Nginx to use them.

volumes:
 - name: token
   emptyDir:
     medium: Memory
 - name: config
   emptyDir:
     medium: Memory

Both config-generator and token-generator are simple, but they need to listen to some port to prevent the container from falling, so a simple HTTP server is started.

# Start a simple HTTP server for keep container
mkdir -p /server
cd /server
echo "I'm alive" > index.html
python3 -m http.server 8082

Finally, you can deploy using gcloud run services replace proxy.yaml. (Note: Disabling authentication in Cloud Run must be done manually.)

Conclusion

In this article, I introduced how to securely host Cloud Run for each Pull Request. Cloud Run is serverless, so it’s very convenient as it eliminates the need for server management.

The Nginx reverse proxy was fairly complicated, so if there’s a simpler configuration, please let me know!

Special thanks to our English team for their awesome support.

References

• Securely Hosting ‘Flutter on the Web’ and ‘WidgetBook’ Apps on GCS with Individual GitHub Pull Requests
• Certificate Manager overview | Google Cloud
• Authenticating service-to-service | Google Cloud
• Cloud Run で サーバーレス GCS Proxy (Japanese only)
• OpenResty® — Open source
• Cloud Run YAML Reference | Google Cloud
• gcloud auth print-identity-token | Google Cloud CLI Documentation