Building a Robust, Extensible Kubernetes Ingress Architecture

---

In modern cloud-native deployments, exposing multiple services under consistent hostnames (e.g., heunify.com, api.heunify.com, www.heunify.com) while ensuring security, TLS automation, rate-limiting, and future extensibility is critical. Below is a production-ready Kubernetes ingress architecture on AWS (3-node EC2 cluster) designed to meet these requirements.

---

1. Design Requirements & Constraints

We have a 3-node Amazon EC2 Kubernetes cluster (each node untainted, serving as both control-plane and worker). Within this cluster:

• Services to expose

  1. A frontend application (hosted at heunify.com and www.heunify.com).
  2. A backend application (hosted at api.heunify.com).
  3. A MariaDB StatefulSet (3 replicas), which is not directly exposed externally.

• Key objectives

  1. Single, unified external entry point (load balanced across all nodes).

  2. TLS/HTTPS termination using Let’s Encrypt certificates (automatically obtained and renewed).

  3. Rate limiting, IP blacklisting, and logging on ingress traffic.

  4. Hostname and path routing:

     • heunify.com & www.heunify.com → frontend service
     • api.heunify.com → backend service

  5. Future extensibility: new namespaces (e.g., redis, kafka, spark) and new subdomains (e.g., redis.heunify.com, kafka.heunify.com) should slot in seamlessly.

---

2. High-Level Architecture & Components

Below is a conceptual diagram of the final state. All components are Kubernetes-native; two controllers are installed via Helm, plus a handful of YAML resources:

   ┌─────────────────────┐
   │   DNS (Route 53)    │
   │ heunify.com, *.heunify.com │
   └─────────┬───────────┘
             │
             ▼
   ┌─────────────────────┐
   │ AWS ELB (LoadBalancer) │  ← created by the Ingress Controller’s Service
   └─────────┬───────────┘
             │
             ▼
   ┌─────────────────────────────────────────────────────────────────┐
   │ NGINX Ingress Controller (ns: ingress-nginx)                   │
   │ • Watches all Ingress resources across namespaces              │
   │ • Terminates TLS via Secret: heunify-tls-secret                │
   │ • Applies rate limits, blacklist snippets, path-based routing  │
   └─────────┬───────────────────────────────────────────────────────┘
             │
   ┌──────────────────┐   ┌──────────────────┐
   │ frontend-ingress │   │ backend-ingress  │
   │ (ns: frontend)   │   │ (ns: backend)    │
   └───────┬──────────┘   └───────┬──────────┘
           │                      │
           ▼                      ▼
   ┌───────────────┐       ┌───────────────┐
   │ frontend-svc  │       │ backend-svc   │
   │ (ClusterIP)   │       │ (ClusterIP)   │
   └───────┬───────┘       └───────┬───────┘
           │                       │
           ▼                       ▼
   [frontend Pods]           [backend Pods]

Separately:
┌───────────────────────────────────────────────────────────────────┐
│ cert-manager (ns: cert-manager)                                   │
│ • ClusterIssuer: letsencrypt-prod  (ACME HTTP-01 via nginx)       │
│ • Certificate: heunify-tls  (covers heunify.com, www, api)        │
│ • Secret: heunify-tls-secret (mounted by NGINX to serve TLS)      │
└───────────────────────────────────────────────────────────────────┘

Key points:

• Ingress Controller (NGINX) lives in its own namespace (ingress-nginx) and manages a single AWS ELB (LoadBalancer Service).
• Ingress Resources reside in app-specific namespaces (frontend, backend). Each one declares host/path rules, references a shared TLS Secret.
• cert-manager (in namespace cert-manager) uses a ClusterIssuer to request and auto-renew a Certificate covering all relevant DNS names.
• DNS (Route 53, or your preferred provider) points heunify.com, www.heunify.com, and api.heunify.com at the ELB’s external address.

---

3. Step-by-Step Implementation

Two main tasks:

1. Install controllers (ingress-nginx and cert-manager) via Helm.
2. Apply a single YAML file that sets up namespaces, ClusterIssuer, Certificate, and Ingress resources.

3.1. Install Controllers via Helm

3.1.1. Install NGINX Ingress Controller

# 1) Add the ingress-nginx repository
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

# 2) Install into namespace "ingress-nginx"
helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.publishService.enabled=true

• Creates:

  • A Deployment of NGINX Ingress controller pods.
  • A Service of type LoadBalancer, which provisions an AWS ELB with a single external IP or DNS name.

• Use kubectl get svc -n ingress-nginx to retrieve the EXTERNAL-IP or DNS. Point your DNS provider to that value.

3.1.2. Install cert-manager

# 1) Add the Jetstack repository
helm repo add jetstack https://charts.jetstack.io
helm repo update

# 2) Install into namespace "cert-manager"
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --set installCRDs=true

• Deploys cert-manager controller pods and installs CRDs (Issuer, ClusterIssuer, Certificate, etc.).
• Prepares the cluster to automatically request and renew Let’s Encrypt certificates.

---

3.2. Apply the “Mega-Manifest” YAML

Save the following content as mega-ingress-setup.yaml and run:

kubectl apply -f mega-ingress-setup.yaml

################################################################
# 1. Create Namespaces
################################################################
apiVersion: v1
kind: Namespace
metadata:
  name: frontend
---
apiVersion: v1
kind: Namespace
metadata:
  name: backend

################################################################
# 2. ClusterIssuer (Let’s Encrypt via HTTP01 using nginx)
################################################################
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: you@heunify.com                  # Replace with your admin email
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - http01:
          ingress:
            class: nginx                   # Must match ingress.class

---
################################################################
# 3. Certificate covering all hostnames (frontend & API)
################################################################
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: heunify-tls
  namespace: frontend                       # TLS Secret lives in "frontend"
spec:
  secretName: heunify-tls-secret            # Name of the Secret for TLS
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName: heunify.com
  dnsNames:
    - heunify.com
    - www.heunify.com
    - api.heunify.com                       # Include API hostname here

---
################################################################
# 4. Frontend Ingress (namespace: frontend)
################################################################
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: frontend-ingress
  namespace: frontend
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/limit-connections: "20"
    nginx.ingress.kubernetes.io/limit-rps: "10"
    nginx.ingress.kubernetes.io/whitelist-source-range: "0.0.0.0/0"
    nginx.ingress.kubernetes.io/server-snippet: |
      location ~* ^/(wp-admin|config|user) {
        return 444;
      }
spec:
  tls:
    - hosts:
        - heunify.com
        - www.heunify.com
      secretName: heunify-tls-secret
  rules:
    - host: heunify.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: frontend-svc
                port:
                  number: 80
    - host: www.heunify.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: frontend-svc
                port:
                  number: 80

---
################################################################
# 5. Backend (API) Ingress (namespace: backend)
################################################################
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: backend-ingress
  namespace: backend
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/limit-connections: "20"
    nginx.ingress.kubernetes.io/limit-rps: "10"
    nginx.ingress.kubernetes.io/whitelist-source-range: "0.0.0.0/0"
spec:
  tls:
    - hosts:
        - api.heunify.com
      secretName: heunify-tls-secret
  rules:
    - host: api.heunify.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: backend-svc
                port:
                  number: 80

What this manifest does:

1. Namespaces

   • Creates frontend and backend for separation of resources.

2. ClusterIssuer

   • letsencrypt-prod (ACME HTTP-01 via nginx). Cert-manager uses it to obtain/renew Let’s Encrypt certificates for any namespace.

3. Certificate

   • heunify-tls in frontend namespace covers heunify.com, www.heunify.com, and api.heunify.com.
   • When issued, it stores TLS cert+key in the Secret heunify-tls-secret.

4. Ingress: frontend-ingress

   • In namespace frontend.
   • Annotations for rate-limiting (20 connections, 10 req/sec) and a snippet to drop requests probing /wp-admin, /config, /user.
   • TLS references the shared heunify-tls-secret (contains the Let’s Encrypt certificate).
   • Routes traffic for heunify.com and www.heunify.com to frontend-svc:80.

5. Ingress: backend-ingress

   • In namespace backend.
   • Same rate-limit annotations.
   • Routes api.heunify.com to backend-svc:80, using the same heunify-tls-secret.

---

4. Detailed Component Explanations

4.1. NGINX Ingress Controller

• Deployment & Service

  • Installed via Helm in namespace ingress-nginx.
  • Creates a Deployment (NGINX pods) and a Service of type LoadBalancer.
  • AWS automatically provisions an ELB (Classic or Network LB) for that Service.

• Ingress Class

  • Each Ingress resource must set kubernetes.io/ingress.class: "nginx". The controller watches only resources matching its class.

• Dynamic Configuration

  • The controller continuously monitors Ingress objects cluster-wide and regenerates NGINX configuration as needed.
  • On any Ingress create/update/delete, NGINX is reloaded with the new rules.

• Rate Limiting & IP Whitelisting

  • nginx.ingress.kubernetes.io/limit-connections: "20" enforces a max of 20 concurrent connections per client IP.
  • nginx.ingress.kubernetes.io/limit-rps: "10" enforces a max of 10 requests per second per client IP.
  • nginx.ingress.kubernetes.io/whitelist-source-range: "0.0.0.0/0" allows all IPs (adjust as needed to restrict access).

• Custom Snippets

  • Using nginx.ingress.kubernetes.io/server-snippet, you can inject raw NGINX directives. For example, blocking requests matching regex paths (e.g., /wp-admin, /config, /user) by returning 444 (immediate connection drop).

---

4.2. cert-manager & ACME

• cert-manager

  • An in-cluster controller that automates issuance and renewal of X.509 certificates (Let’s Encrypt, Vault, etc.).
  • Relies on ACME (Automatic Certificate Management Environment) to interact with Let’s Encrypt.

• ClusterIssuer

  • A cluster-scoped resource named letsencrypt-prod.

  • Configured to use ACME HTTP-01 challenge via ingress.class: nginx.

  • When a Certificate references this issuer, cert-manager orchestrates the ACME HTTP-01 flow:

    1. Creates a temporary Challenge resource.
    2. Patches an Ingress to serve a challenge token under /.well-known/acme-challenge/....
    3. Let’s Encrypt validates the challenge by requesting that path on heunify.com (through the NGINX Ingress).
    4. When validation succeeds, cert-manager stores the resulting certificate and private key in a Secret.

• Certificate Resource

  • Declares commonName and dnsNames. In this setup, it includes heunify.com, www.heunify.com, and api.heunify.com.
  • References the ClusterIssuer.
  • On creation, cert-manager spawns a Challenge → NGINX serves it → Let’s Encrypt validates → Secret heunify-tls-secret is populated.
  • Renewal: cert-manager automatically re-runs the ACME flow ~30 days before expiry, updating the Secret without manual intervention.

---

4.3. Ingress Resources

4.3.1. Frontend Ingress (Namespace: frontend)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: frontend-ingress
  namespace: frontend
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/limit-connections: "20"
    nginx.ingress.kubernetes.io/limit-rps: "10"
    nginx.ingress.kubernetes.io/whitelist-source-range: "0.0.0.0/0"
    nginx.ingress.kubernetes.io/server-snippet: |
      location ~* ^/(wp-admin|config|user) {
        return 444;
      }
spec:
  tls:
    - hosts:
        - heunify.com
        - www.heunify.com
      secretName: heunify-tls-secret
  rules:
    - host: heunify.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: frontend-svc
                port:
                  number: 80
    - host: www.heunify.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: frontend-svc
                port:
                  number: 80

• Namespace: Keeps the Ingress resource close to the Services it routes (frontend).

• Annotations:

  • ingress.class: "nginx" ensures the NGINX controller picks it up.
  • limit-connections, limit-rps, whitelist-source-range enforce per-IP rate limiting and allow-all IPs by default.
  • server-snippet blocks common attack probes (e.g., /wp-admin, /config, /user) by returning 444.

• TLS: Uses the shared Secret heunify-tls-secret (populated by cert-manager) for both heunify.com and www.heunify.com.

• Rules: Two rules entries for heunify.com and www.heunify.com, each routing / to frontend-svc:80.

4.3.2. Backend (API) Ingress (Namespace: backend)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: backend-ingress
  namespace: backend
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/limit-connections: "20"
    nginx.ingress.kubernetes.io/limit-rps: "10"
    nginx.ingress.kubernetes.io/whitelist-source-range: "0.0.0.0/0"
spec:
  tls:
    - hosts:
        - api.heunify.com
      secretName: heunify-tls-secret
  rules:
    - host: api.heunify.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: backend-svc
                port:
                  number: 80

• Shares the same TLS Secret (heunify-tls-secret) as the frontend Ingress, covering api.heunify.com.
• Rate-limit annotations match those in frontend.
• Rule: Routes all traffic for api.heunify.com to backend-svc:80.

Why include api.heunify.com in the Certificate? Without that DNS name in the dnsNames list, TLS handshakes for api.heunify.com would fail. Bundling all hostnames into one multi-SAN certificate reduces complexity and stays well within Let’s Encrypt’s limit (100 names per certificate).

---

5. Security & Operational Best Practices

1. Rate Limiting

   • Prevents denial-of-service from a single IP.
   • limit-connections: 20 ensures a client can’t open more than 20 concurrent connections.
   • limit-rps: 10 ensures no more than 10 requests/sec per IP.

2. IP Blacklisting / Custom Snippets

   • The server-snippet directive drops requests matching regexes (e.g., /wp-admin).
   • For more advanced application-layer filtering, consider adding a WAF (ModSecurity) sidecar.

3. fail2ban / Node-Level Protection

   • Deploy a DaemonSet that tails NGINX access logs on each node.
   • If an IP repeatedly triggers suspicious patterns (e.g., probing non-existent .php files), fail2ban can automatically ban it via iptables at the node level.

4. TLS & Let’s Encrypt Automation

   • ACME HTTP-01 is straightforward, since the NGINX controller already serves port 80.
   • cert-manager handles renewals ~30 days before expiration.
   • The TLS Secret (heunify-tls-secret) is updated transparently; NGINX reloads on secret change.

5. Namespace Isolation

   • The Ingress Controller and cert-manager reside in dedicated namespaces (ingress-nginx, cert-manager).
   • Application namespaces (frontend, backend, etc.) remain focused on business logic.
   • RBAC boundaries ensure the controller only needs read access to Services/Secrets, while apps don’t modify controller configuration.

6. DNS Configuration

   • After Helm install, run:

     kubectl get svc ingress-nginx-controller -n ingress-nginx

   • Note the EXTERNAL-IP (AWS ELB DNS).

   • In Route 53 (or your provider), create:

     heunify.com      A (alias) → <ELB DNS>
     www.heunify.com  A (alias) → <ELB DNS>
     api.heunify.com  CNAME      → <ELB DNS>

7. IP Whitelisting (Optional)

   • Currently set to allow all (0.0.0.0/0).

   • To restrict, replace with your corporate network:

     nginx.ingress.kubernetes.io/whitelist-source-range: "203.0.113.0/24,198.51.100.0/24"

8. Monitoring & Logging

   • Ingress-nginx can export Prometheus metrics.
   • Forward access logs to a central system (e.g., Elasticsearch/Fluentd/Kibana stack or CloudWatch).
   • Integrate with fail2ban for automated IP bans.

---

6. Extensibility for Future Services

Adding new services or subdomains follows the same pattern:

1. Create a new namespace, e.g., redis.

2. Deploy your Service (e.g., redis-svc:6379).

3. Create a new Ingress resource in namespace redis, referencing heunify-tls-secret. Example:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: redis-ingress
     namespace: redis
     annotations:
       kubernetes.io/ingress.class: "nginx"
   spec:
     tls:
       - hosts:
           - redis.heunify.com
         secretName: heunify-tls-secret
     rules:
       - host: redis.heunify.com
         http:
           paths:
             - path: /
               pathType: Prefix
               backend:
                 service:
                   name: redis-svc
                   port:
                     number: 6379

4. Add DNS record

   redis.heunify.com  CNAME → <ELB DNS>

If desired, you can create individual Certificates per-namespace rather than one mega-certificate. Cert-manager allows that by referencing the same ClusterIssuer in each namespace. However, a multi-SAN certificate is simpler until you hit Let’s Encrypt’s limit (100 names per certificate).

---

7. Conclusion

This architecture combines:

• NGINX Ingress Controller (Helm-managed, AWS ELB fronted)
• cert-manager (ClusterIssuer + Certificate for automation)
• Namespace-scoped Ingress resources with rate-limits, blacklists, path-based routing
• Single AWS ELB as the public entry point

Result:

• A single unified entry point (ELB DNS) for all HTTP/HTTPS traffic.
• Automatic Let’s Encrypt TLS with zero manual renewals.
• Granular, per-host, per-path routing to multiple backend services.
• Built-in security: connection limits, request rate limits, IP blocking for known attack patterns.
• Horizontal scalability: add more Ingress resources in new namespaces for future microservices, all served by the same NGINX controller and certificate.

By following these principles, you gain a secure, maintainable, and highly available ingress layer on Kubernetes.