How to Securely Host Cloud Run for Each Pull Request Using Identity-Aware Proxy
Hello, I’m riddle from the SRE Group at MIXI Development Division.
Previously, I wrote an article about Securely Hosting ‘Flutter on the Web’ and ‘WidgetBook’ Apps on GCS with Individual GitHub Pull Requests.This time I’ll try something similar with Cloud Run.
Hosting Cloud Run for each Pull Request is simple, but using Identity-Aware Proxy can be tricky, so I’ll focus on that.
Table of Contents:
- Structure Creation Overview
- How It Works
2.1 Developer Accesses https://XXXXX.example.com
2.2 Identity-Aware Proxy Authenticates the Developer
2.3 Nginx Reverse Proxy to Cloud Run
2.4 Request to Cloud Run - Cloud Run YAML File
- Conclusion
- References
Structure Creation Overview
The flow when creating a Pull Request (PR) is as follows:
- A PR is created on GitHub.
- GitHub Actions deploy Cloud Run, obtaining the URL https://XXXXX.a.run.app from Cloud Run.
- GitHub Actions add an A record for
XXXXX.example.com
using the IP address used by the LoadBalancer (LB).
Access works as follows:
- Developer accesses https://XXXXX.example.com.
- Identity-Aware Proxy authenticates/authorizes the Developer.
- If authentication/authorization is successful, access is granted.
How It Works
I’ll explain the communication flowchart below.
1. Developer Accesses https://XXXXX.example.com
Since GitHub Actions have registered an A record, when the Developer accesses https://XXXXX.example.com, it reaches the LB. SSL Termination is performed at the LB for HTTPS communication.
Google Cloud’s Certificate Manager can issue wildcard certificates, so we use it to issue certificates for *.example.com. This allows certificate verification for any subdomain (e.g., https://hogehoge.example.com).
2. Identity-Aware Proxy Authenticates the Developer
Using the Identity-Aware Proxy, we can authenticate/authorize the Developer. Therefore, we grant httpsResourceAccessor
permission to the Google account of the Developer we want to allow access.
3. Nginx Reverse Proxy to Cloud Run
Once Authentication/Authorization is successful through the Identity-Aware Proxy, Nginx is used to reverse proxy to Cloud Run. Here, Nginx reverse proxies to https://XXXXX.run.app. This part is fairly complicated.
Here is the kind of config I wrote:
# default Cloud Run Resolver is 169.254.1.1
# but this resolver does not work on Cloud Run to resolve subdomain.a.run.app
resolver 8.8.8.8;
server {
listen 80;
server_name ~^(?.+)\.example\.com$;
location / {
# set Authorization header by /token/auth_token
rewrite_by_lua_block {
local file = io.open("/token/auth_token")
io.input(file)
local data = io.read()
io.close()
ngx.req.set_header("Authorization", "Bearer "..data);
local subdomain = ngx.var.subdomain
}
# forwarding
proxy_pass https://$subdomain.a.run.app;
proxy_set_header Host $subdomain.a.run.app;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# remove X-Serverless-Authorization header
# IAP add requester account's OIDC ID Token to this header
# and this header use by Google Front End instead of Authorization header
proxy_set_header X-Serverless-Authorization "";
}
}
Some points to note:
The default DNS resolver set in Cloud Run cannot resolve subdomain.a.run.app
, so I point the resolver externally.
# default Cloud Run Resolver is 169.254.1.1
# but this resolver does not work on Cloud Run to resolve subdomain.a.run.app
resolver 8.8.8.8;
When accessing Cloud Run, it’s necessary to attach the ID Token of an account with Cloud Run Invoker to the Authorization header. Therefore, I write the ID Token to /token/auth_token and set it in the Nginx Authorization header.
※Inspired by: https://zenn.dev/shin5ok/articles/abe99ad833aeee#nginx%E3%81%AE%E8%A8%AD%E5%AE%9A
# set Authorization header by /token/auth_token
rewrite_by_lua_block {
local file = io.open("/token/auth_token")
io.input(file)
local data = io.read()
io.close()
ngx.req.set_header("Authorization", "Bearer "..data);
local subdomain = ngx.var.subdomain
}
The X-Serverless-Authorization
header is added by IAP when sending a request to Cloud Run (Authenticating service-to-service | Google Cloud).
This header is prioritized by Google Front End over the Authorization header, so I remove it.
# remove X-Serverless-Authorization header
# IAP add requester account's OIDC ID Token to this header
# and this header use by Google Front End instead of Authorization header
proxy_set_header X-Serverless-Authorization "";
4. Request to Cloud Run
Cloud Run, which is publicly accessible and requires authentication, is normally inaccessible, but can be accessed by Nginx attaching the Authorization header to the request.
This allows Developers to access Cloud Run.
Cloud Run YAML File
Here is the actual Cloud Run YAML file I used:
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
annotations:
run.googleapis.com/ingress: internal-and-cloud-load-balancing
run.googleapis.com/launch-stage: BETA
labels:
cloud.googleapis.com/location: asia-northeast1
name: pr-cloudrun-with-iap
spec:
template:
metadata:
annotations:
autoscaling.knative.dev/maxScale: "1"
run.googleapis.com/container-dependencies: '{"proxy":["config-generator", "token-generator"]}'
spec:
# add Cloud Run Invoker Role / Storage Object Viewer Role
serviceAccountName: pr-cloudrun@[YOUR_PROJECT].iam.gserviceaccount.com
containerConcurrency: 80
containers:
- name: proxy
image: openresty/openresty:latest
ports:
- containerPort: 80
name: http1
resources:
limits:
cpu: 1000m
memory: 512Mi
startupProbe:
tcpSocket:
port: 80
volumeMounts:
- name: token
readOnly: true
mountPath: /token
- name: config
readOnly: true
mountPath: /etc/nginx/conf.d
- name: config-generator
image: gcr.io/google.com/cloudsdktool/google-cloud-cli:slim
command: ["/bin/sh"]
args:
- "-c"
- |
cat << "EOF" > /config/proxy.conf
# default Cloud Run Resolver is 169.254.1.1
# should be changed to resolve subdomain.a.run.app
resolver 8.8.8.8;
server {
listen 80;
server_name ~^(?.+)\.example\.com$;
location / {
# set Authorization header by /token/auth_token
rewrite_by_lua_block {
local file = io.open("/token/auth_token")
io.input(file)
local data = io.read()
io.close()
ngx.req.set_header("Authorization", "Bearer "..data);
local subdomain = ngx.var.subdomain
}
# forwarding
proxy_pass https://$subdomain.a.run.app;
proxy_set_header Host $subdomain.a.run.app;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# remove X-Serverless-Authorization header
# IAP add requester account's OIDC ID Token to this header
# and this header use by Google Front End instead of Authorization header
proxy_set_header X-Serverless-Authorization "";
}
}
EOF
# Start a simple HTTP server for keep container
mkdir -p /server
cd /server
echo "I'm alive" > index.html
python3 -m http.server 8081
# / is redirect to other cloud run service
# so, use tcp probe instead of http probe
startupProbe:
tcpSocket:
port: 8081
volumeMounts:
- name: config
readOnly: true
mountPath: /config
- name: token-generator
image: gcr.io/google.com/cloudsdktool/google-cloud-cli:slim
command: ["/bin/sh"]
args:
- "-c"
- |
# create first, and avoid no file error
gcloud auth print-identity-token > /token/auth_token
# Start a simple HTTP server for keep container
mkdir -p /server
cd /server
echo "I'm alive" > index.html
python3 -m http.server 8082
# update token every 3500 seconds
# because token is expired after 3600 seconds
while true; do gcloud auth print-identity-token > /token/auth_token ; sleep 3500; done
startupProbe:
tcpSocket:
port: 8082
volumeMounts:
- name: token
readOnly: true
mountPath: /token
timeoutSeconds: 300
volumes:
- name: token
emptyDir:
medium: Memory
- name: config
emptyDir:
medium: Memory
traffic:
- latestRevision: true
percent: 100
Some points to note:
The Service Account assigned to Cloud Run has the Cloud Run Invoker Role
and Storage Object Viewer Role
. The Cloud Run Invoker Role is for accessing other Cloud Run services.
I also assigned the Storage Object Viewer Role
because it was necessary to obtain the official image from Google Container Registry.
serviceAccountName: pr-cloudrun@[YOUR_PROJECT].iam.gserviceaccount.com
This YAML starts three containers, including two sidecars:
proxy
: Nginx reverse proxy.config-generator
: Container to generate Nginx config.token-generator
: Container to generate ID Token for Cloud Run Invoker.
The proxy
, as explained earlier, uses the OpenResty image, which includes various 3rd party modules for Lua.
The config-generator
is a container for generating Nginx config, as ConfigMap is not available in Cloud Run.
The token-generator
generates the ID Token for Cloud Run connection. The ID Token attached to the Authorization
header earlier needs to have the Cloud Run Invoker Role, so it’s generated using the permissions of the ServiceAccount attached to Cloud Run itself.
The generated ID Token expires after 1 hour, so it’s generated at container startup and then updated with time to spare.
while true; do gcloud auth print-identity-token > /token/auth_token ; sleep 3500; done
Config and ID Token are shared between containers using Volumes, allowing Nginx to use them.
volumes:
- name: token
emptyDir:
medium: Memory
- name: config
emptyDir:
medium: Memory
Both config-generator and token-generator are simple, but they need to listen to some port to prevent the container from falling, so a simple HTTP server is started.
# Start a simple HTTP server for keep container
mkdir -p /server
cd /server
echo "I'm alive" > index.html
python3 -m http.server 8082
Finally, you can deploy using gcloud run services replace proxy.yaml
. (Note: Disabling authentication in Cloud Run must be done manually.)
Conclusion
In this article, I introduced how to securely host Cloud Run for each Pull Request. Cloud Run is serverless, so it’s very convenient as it eliminates the need for server management.
The Nginx reverse proxy was fairly complicated, so if there’s a simpler configuration, please let me know!
Special thanks to our English team for their awesome support.
References
- Securely Hosting ‘Flutter on the Web’ and ‘WidgetBook’ Apps on GCS with Individual GitHub Pull Requests
- Certificate Manager overview | Google Cloud
- Authenticating service-to-service | Google Cloud
- Cloud Run で サーバーレス GCS Proxy (Japanese only)
- OpenResty® — Open source
- Cloud Run YAML Reference | Google Cloud
- gcloud auth print-identity-token | Google Cloud CLI Documentation