kiquetal
System Engineer
LINKEDIN-PROFILE
August 31, 2026
Updated: Sep 7, 2026

SPIFFE on ECS with Envoy

#spiffe #spire #ecs #envoy #aws #security #mtls

I have been studying envoy for a while, because I was very satisfied with istio, but i wanted to know how it works under the hood. The goal was very clear: implement a mTLS solution for ECS, a poor’s version of zero trust.

We create the architecture using:

  • SPIRE Server + Admission Controller to allow/deny regiser in a service discovery
  • SPIRE-Agent + Envoy sidecar to provide mTLS
  • envoy sidecar to provide mTLS between services
  • a custom proteus_ecs node attestor (two plugin halves: agent-side gathers the task claim, server-side verifies it against the ECS API) — because Fargate has no EC2 host for the usual aws_iid attestor

We have configured the envoy to go from service-a to service-b using mtls, for this goal we configured the envoy to use SDS (Secret Discovery Service) to get the SVIDs from SPIRE-Agent, and we configured the SPIRE-Agent to get the SVIDs from SPIRE-Server.

The admission controller is used to allow/deny the registration of the service in the service discovery, this is a very important step to avoid that a malicious service can register in the service discovery and get access to the other services.

Architecture

Identity is established in two layers: first SPIRE decides which task it trusts (node attestation), then the admission controller decides which service identity that task is allowed to get (workload attestation).

Proteus container view — SPIRE Server + Admission Controller task, and two service tasks each running app, Envoy sidecar (egress :9903 / ingress :9902) and SPIRE Agent

Node attestation — proving the ECS task

On Fargate there is no EC2 instance, so the usual aws_iid node attestor doesn’t apply. I wrote a small custom node attestor, proteus_ecs — and like every SPIRE node attestor it comes in two halves: an agent-side plugin that gathers and sends the claim, and a server-side plugin that verifies it and decides whether to issue the node SVID. This runs automatically when the SPIRE Agent starts — SPIRE invokes the plugin as part of the agent’s boot handshake; nothing triggers it manually. The agent side reads the task metadata endpoint and forwards the task ARN, cluster and family:

metadataURI := os.Getenv("ECS_CONTAINER_METADATA_URI_V4")
body, _ := httpGet(metadataURI + "/task")   // TaskARN, Cluster, Family

The important part is on the server side: it never trusts the payload blindly. It calls the ECS API to confirm the task and checks the task’s IAM role against an allow-list:

taskInfo, _ := describeTask(ctx, region, cluster, payload.TaskARN) // ECS DescribeTasks
if !isRoleAllowed(taskInfo.TaskRoleARN) {
    return status.Error(codes.PermissionDenied, "task role not allowed")
}
// → spiffe://proteus.local/agent/ecs/<task-id>
//   selectors: ecs:cluster:*, ecs:family:*, ecs:task-role:*

So a task can claim an ARN, but the server verifies it out-of-band via DescribeTasks and only accepts pre-approved IAM roles.

Node attestation sequence — the SPIRE Agent claims a task ARN; the SPIRE Server verifies it against the ECS API and an IAM-role allow-list before issuing a node SVID

How to read it: the SPIRE Agent only claims an identity (it forwards the task ARN it read from metadata). The SPIRE Server is the trust anchor — it calls the ECS API (DescribeTasks) to confirm the task really exists and to fetch its IAM role, then checks that role against an allow-list. Only if the role passes does it mint the node SVID. This is why a compromised task can’t forge an identity: the proof comes from AWS, not from the agent’s own claim.

Workload attestation + admission — proving the service

Node attestation says which task; the admission controller says which service identity that task may hold. On POST /admit it creates a SPIRE registration entry via the Entry API:

spiffeID  := "spiffe://" + trustDomain + "/" + serviceName // e.g. .../service-a
selectors := []string{"unix:uid:1000"}
entryID, _ := spire.CreateEntry(ctx, parentID, spiffeID, selectors)
log.Printf("ADMITTED: service=%s spire_entry=%s", serviceName, entryID)

Until that entry exists, the agent has nothing to match and SDS returns workload is not authorized — this is the deny-by-default behavior the test below demonstrates. (The workload selector here is a coarse unix:uid:1000; tighter selectors are a natural next step.)

Admission sequence — the admission controller creates a SPIRE registration entry so the agent's SDS request can be matched and a workload SVID issued

How the two layers relate. The proteus_ecs plugin and the admission controller answer two different questions, and a workload only gets a usable identity when both pass:

  • proteus_ecs plugin (node attestation) — is this really the ECS task it claims to be? Its server-side half calls the ECS API (DescribeTasks) to verify the task and its IAM role, then issues a node SVID (spiffe://proteus.local/agent/ecs/<task-id>). Trust comes from AWS, not from the workload’s own claim.
  • Admission controller (workload attestation) — is this task allowed to hold this service identity? On POST /admit it creates a registration entry whose parentID is exactly that node SVID. That parent-child link is the seam: the controller can only grant a workload identity to a node the plugin already vouched for.

How the entry finds its agent. The CreateEntry call carries three fields that matter here:

CreateEntry(ctx, parentID, spiffeID, selectors)
// parentID  = spiffe://proteus.local/agent/ecs/<task-id>  ← the agent's node SVID
// spiffeID  = spiffe://proteus.local/service-a            ← the workload identity
// selectors = [unix:uid:1000]                             ← which process on that node

When Envoy later asks its local agent for service-a’s SVID over SDS, the SPIRE Server decides whether to issue it by matching two things at once:

  1. parentID — is the requesting agent’s node SVID equal to the entry’s parentID? This binds the entry to one specific agent. service-a’s entry can only ever be served by the agent whose node SVID is spiffe://proteus.local/agent/ecs/<task-id> — i.e. the agent inside service-a’s own task. No other task’s agent matches.
  2. selectors — does the calling workload satisfy the selectors (unix:uid:1000)? This narrows it to the right process within that node.

Only when both match does the Server mint and stream the SVID. This is the security property: even if an attacker knew service-a’s SPIFFE ID, they couldn’t obtain its SVID from a different task, because their agent’s node identity wouldn’t equal the entry’s parentID. And it’s why node attestation must run first — without a node SVID there is no parentID to point the entry at.

So the plugin establishes infrastructure trust (a genuine ECS task with an approved role); the admission controller layers application policy on top (this task may be service-a). Neither alone is enough — the SVID is minted only when the verified node has an admitted entry to match.

All three control-plane hops run over gRPC: the attestor plugin (streaming RPC), the admission controller’s CreateEntry (SPIRE Entry API), and Envoy’s SDS (gRPC/HTTP2) — the SPIRE-local ones over a Unix socket.

Testing the architecture

The setup is intentionally minimal: two ECS services, service-a (the caller) and service-b (the callee), each with an Envoy sidecar and a co-located SPIRE Agent. service-a calls service-b over mTLS on /api/data. Both proxies fetch their identities from SPIRE via SDS.

The test walks through a dark → admit → live progression, proving that no workload receives an identity until it has been explicitly admitted.

Dark by default vs live after admit — SDS denies the SVID until the admission controller creates the SPIRE entry

The same story as a sequence: in the dark branch Envoy’s SDS request is denied and the call fails with 503 UF; in the live branch — after the admission controller creates the SPIRE entry — the SVID is pushed and the mTLS call to service-b succeeds.

Proteus mTLS sequence — dark (SDS denied, 503 UF) vs live (SVID pushed, mTLS succeeds) after admission

Step 1 — Dark state: no workload is admitted yet

Before any admission, Envoy asks SPIRE for an SVID and is rejected. The Envoy log on service-a shows the request failing with 503 flags=UF and, underneath, the SDS stream being closed:

workload is not authorized for the requested identities ["spiffe://proteus.local/service-a"]

Envoy on service-a denied — workload not authorized, 503 UF

On the service-b side, the SPIRE Agent reports the same thing from its own perspective — an SDS StreamSecrets request for spiffe://proteus.local/service-b fails with InvalidArgument: workload is not authorized:

SPIRE Agent on service-b denied — error building stream secrets

This is the whole point of the design: identity is deny-by-default.

What these logs actually prove. The workload is not authorized errors are a workload attestation failure — there’s no registration entry matching the requested SPIFFE ID yet. They are not node attestation. In fact, for the agent to reach this point it must have already node-attested successfully at startup (otherwise it couldn’t talk to the Server at all). Node attestation happens first, almost immediately when the task boots; it just isn’t what these dark-state screenshots capture. To see it directly you’d look at the agent/server startup logs where the node SVID (spiffe://proteus.local/agent/ecs/<task-id>) is issued.

Step 2 — Admit service-a

The Admission Controller is what flips a workload from dark to allowed. Once it approves service-a, it logs the admission and the SPIRE registration entry it created:

ADMITTED: microvm=service-a-task service=service-a spire_entry=6f842eda-9f2b-4f68-9bb8-ed514ec07937

Admission controller admits service-a and creates the SPIRE entry

Step 3 — service-a receives its SVID

With the entry in place, the SPIRE Agent on service-a creates the workload entry, mints the X.509 SVID, and streams it to Envoy over SDS:

Entry created ... spiffe_id="spiffe://proteus.local/service-a" → Creating X509-SVID → Sending StreamSecrets response

SPIRE Agent on service-a creating the X509-SVID and streaming it via SDS

Step 4 — Admit service-b and issue its SVID

The same flow repeats for service-b. After it’s admitted, its SPIRE Agent creates the entry, updates the SVID, and serves it to Envoy:

Entry created ... spiffe://proteus.local/service-b → Creating X509-SVID → SVID updated

SPIRE Agent on service-b creating and updating the X509-SVID after admission

Step 5 — Live mTLS call succeeds

With both workloads holding valid SVIDs, the mTLS call goes through end to end.

On service-a, Envoy logs the outbound call succeeding:

[mTLS-OUT] GET /api/data 200 upstream=10.0.0.141:9902

Envoy on service-a — successful mTLS-OUT call to service-b, HTTP 200

On service-b, Envoy logs the inbound call and — crucially — the verified peer identity from the client certificate:

[mTLS-IN] GET /api/data 200 peer=spiffe://proteus.local/service-a

Envoy on service-b — inbound mTLS with verified peer SPIFFE ID service-a

The peer=spiffe://proteus.local/service-a field is the payoff: service-b didn’t just accept a TLS connection, it cryptographically verified who was calling — no shared secret, no static cert, identity issued and rotated by SPIRE.

Conclusion

This hands-on project showed me the value of a control plane like Istio. The key lesson: communication should only be allowed between the listeners explicitly defined in each Envoy’s configuration. The hard part is doing that dynamically — when you launch many services, something has to create and update the Envoy configuration every time a new service joins. That “something” is exactly the job of a control plane, or of a purpose-built component that reconciles Envoy config as the service topology changes. Building it by hand made that responsibility concrete in a way that using Istio never did.

The piece I wasn’t familiar with going in was the port hop. The app never speaks mTLS or even touches the network directly — it makes a plain HTTP call to its own sidecar on localhost:9903 (the egress listener), and Envoy transparently negotiates mTLS to the peer’s :9902 ingress listener. So “the app calls localhost and mutual TLS just happens” is the mental model that took me a while to internalize. The trust boundary stops being where a service is (an IP or security group) and becomes who it is — service-b accepts the call because it cryptographically verified peer=spiffe://proteus.local/service-a, not because of a network rule.

Next steps I’d tackle: tighter workload selectors than the coarse unix:uid:1000, an automated admission trigger instead of a manual POST /admit, and a comparison against AWS App Mesh’s native mTLS to see what the from-scratch approach buys versus the managed path.

Proteus C4 System Context — SPIFFE on ECS with Envoy

SYSTEM BROADCASTS

Subscribe to receive telemetry logs directly in your inbox upon new entry compiles.

ADDR_
Lat: 25.2637° S
Lon: 57.5759° W
Node_Primary: Active Uptime: 99.999%