Building a Self-Managed CI/CD Pipeline on AWS EC2 — Without ECR, Without EKS

At Ohwise, we believe in controlling our own infrastructure without giving up automation. This guide shows how to use GitHub Actions and the AWS Systems Manager (SSM) Agent to continuously deploy a Dockerized Kubernetes app on plain EC2 servers — no EKS, no ECR, and no hidden magic. It’s fast, auditable, and nearly zero-cost — proving that you don’t need managed services to achieve enterprise-grade automation.

---

🎯 The Goal: A Lean but Reliable CI/CD Pipeline

Most modern CI/CD tutorials assume you’re on EKS (Elastic Kubernetes Service) and storing images in ECR (Elastic Container Registry). But many teams — especially startups, indie founders, or advanced engineers — prefer to run Kubernetes directly on EC2 for cost control, flexibility, and insight into what’s really happening under the hood.

Our objective:

• ✅ Automatically build a Docker image when you git push origin main
• ✅ Push it to Docker Hub, our external container registry
• ✅ Use AWS SSM to remotely trigger a Kubernetes rolling update on our EC2 instance
• ✅ Do it all securely, without SSH access or exposed ports

The workflow ends with a fresh deployment of your app — all from a single push.

---

🧩 1. Setting Up GitHub Secrets

Secrets are the bridge between your GitHub Action runner and the services it touches.

Go to GitHub → Repo → Settings → Secrets → Actions and create these:

Always treat these secrets as production credentials — never echo them to logs.

---

🔐 2. IAM Roles and Permissions

Proper IAM configuration is what makes this setup secure. We use two distinct roles:

1. The EC2 instance role for the SSM Agent (so AWS can run commands on the node).
2. The GitHub OIDC role for GitHub Actions (so the workflow can call SSM APIs).

---

🖥️ a. EC2 Instance Role — for SSM Agent

Purpose: allow the instance to register with Systems Manager and receive commands.

Steps:

1. In IAM → Roles → Create role → AWS service → EC2
2. Attach the managed policy AmazonSSMManagedInstanceCore
3. Name it EC2_SSM_Access_Role
4. Attach it to your instance (EC2 → Instance → Actions → Security → Modify IAM role)

That’s it — your node can now communicate with SSM.

---

☁️ b. GitHub OIDC Role — for the Pipeline

Purpose: give GitHub Actions temporary credentials via OpenID Connect (OIDC). This avoids long-lived AWS keys in your repo.

1. Create Role → Web Identity

   • Provider: token.actions.githubusercontent.com
   • Audience: sts.amazonaws.com

2. Add a condition restricting the repository and branch:

   "StringLike": {
     "token.actions.githubusercontent.com:sub": "repo:<user>/<repo>:ref:refs/heads/main"
   }

3. Attach a minimal inline policy:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "ssm:SendCommand",
           "ssm:GetCommandInvocation",
           "ssm:ListCommandInvocations",
           "ec2:DescribeInstances"
         ],
         "Resource": "*"
       }
     ]
   }

4. Copy its ARN → store as AWS_ROLE_ARN secret.

This separation of roles ensures GitHub can request the rollout, but only the EC2 node can perform it.

---

🧰 3. Installing and Testing the SSM Agent on Ubuntu 22.04

Your EC2 node must be “SSM-managed,” meaning it’s running the AWS SSM Agent.

# Remove old snap version if present
sudo snap remove amazon-ssm-agent 2>/dev/null || true

# Install the correct Ubuntu 22.04 package
curl -o /tmp/amazon-ssm-agent.deb \
  https://s3.us-east-1.amazonaws.com/amazon-ssm-us-east-1/latest/ubuntu_22.04_amd64/amazon-ssm-agent.deb
sudo dpkg -i /tmp/amazon-ssm-agent.deb

sudo systemctl enable --now amazon-ssm-agent
sudo systemctl status amazon-ssm-agent

Check registration:

aws ssm describe-instance-information --region us-east-1 --output table

You should see your instance listed as Online.

If not, inspect logs:

sudo journalctl -u amazon-ssm-agent -e --no-pager

✅ Tip: the agent needs outbound internet or VPC endpoints:

• com.amazonaws.<region>.ssm
• ec2messages
• ssmmessages

Once “Online,” your pipeline can execute remote commands.

---

🧠 4. Why the SSM Agent Matters

The AWS SSM Agent acts like a secure tunnel. It lets you run shell commands on your EC2 node without SSH, without open ports, and with full audit logging in CloudTrail.

In our setup, SSM is how GitHub Actions remotely executes:

kubectl -n ohwise set image deployment/ohwise-web-hub ...

Think of it as a “serverless SSH,” purpose-built for automation.

---

🧾 Ensure Kubernetes Access for root

Because SSM runs as root, root must be able to use kubectl.

sudo mkdir -p /root/.kube
sudo cp /etc/kubernetes/admin.conf /root/.kube/config
sudo chmod 600 /root/.kube/config

If you skip this, kubectl defaults to http://localhost:8080, which fails inside automation.

---

🏷️ 5. Choosing a Tagging Strategy for Docker Images

Tagging defines how Kubernetes decides whether an image is new.

For production, prefer immutable tags — either timestamps or SHAs. If you truly need latest, ensure your deployment uses:

imagePullPolicy: Always

Otherwise the pods will not pull new layers, and your rollout will appear successful but run stale code.

---

⚙️ 6. Full GitHub Actions Workflow

Below is the full working workflow Ohwise uses.

name: Build Docker Image and Deploy to EC2
on:
  push:
    branches: [ "main" ]

permissions:
  contents: write
  id-token: write

env:
  IMAGE_NAME: liveget/ohwise-web-hub
  K8S_NAMESPACE: ohwise
  K8S_DEPLOYMENT: ohwise-web-hub
  K8S_CONTAINER: ohwise-web-hub
  AWS_REGION: us-east-1

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4

    - name: Build the Docker image
      run: |
        TAG=$(date +%s)
        echo "TAG=$TAG" >> $GITHUB_ENV
        echo "Building image: $IMAGE_NAME:$TAG"
        docker build -t $IMAGE_NAME:$TAG .

    - name: Login to Docker Hub
      run: echo "${{ secrets.DOCKERHUB_TOKEN }}" | docker login -u "${{ secrets.DOCKERHUB_USERNAME }}" --password-stdin

    - name: Push Docker image
      run: docker push $IMAGE_NAME:$TAG

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v4
      with:
        role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
        aws-region: ${{ env.AWS_REGION }}

    - name: Trigger rollout via SSM
      run: |
        aws ssm send-command \
          --instance-ids "${{ secrets.SSM_INSTANCE_ID }}" \
          --document-name "AWS-RunShellScript" \
          --parameters '{"commands":[
            "bash -c \"set -eux; export HOME=/root; export KUBECONFIG=/root/.kube/config; export PATH=/usr/local/bin:/usr/bin:/bin:/snap/bin:$PATH; kubectl -n '"$K8S_NAMESPACE"' set image deployment/'"$K8S_DEPLOYMENT"' '"$K8S_CONTAINER"'='"$IMAGE_NAME:$TAG"' --record; kubectl -n '"$K8S_NAMESPACE"' rollout status deployment/'"$K8S_DEPLOYMENT"' --timeout=300s\""
          ]}'

This workflow runs cleanly on GitHub-hosted Ubuntu runners, builds the image, uploads it to Docker Hub, assumes the AWS OIDC role, and triggers a rolling deployment via SSM.

---

💡 7. Lessons Learned and Important Details

🧩 7.1 pipefail is not supported in /bin/sh

By default, AWS-RunShellScript runs commands with /bin/sh (which is dash on Ubuntu). Dash is fast and POSIX-compliant but does not support pipefail.

That’s why our command wraps everything in:

bash -c "set -eux ..."

so Bash interprets the script. If you omit that wrapper, you’ll get:

set: Illegal option -o pipefail

---

⚙️ 7.2 Why kubectl rollout status is the right approach

Many engineers simply run:

kubectl apply -f deployment.yml

and call it done. But apply doesn’t guarantee pods are healthy — it returns immediately.

kubectl rollout status waits for new pods to become Ready or fails if they don’t within the timeout. This gives CI a definitive success/failure signal and prevents silently broken deployments. It’s the production-safe way to automate updates.

---

🧰 7.3 Why AWS-RunShellScript

• Already available in every region — no custom SSM Document required
• Executes arbitrary Bash commands under root
• Fully audited in AWS CloudTrail
• Works on any instance with the SSM Agent installed

It’s effectively your programmable SSH replacement.

---

🏠 7.4 Why HOME, KUBECONFIG, and PATH Matter

SSM executes in a non-interactive, root shell that lacks user environment variables.

If you rely on implicit defaults:

• HOME may be empty → no ~/.kube/config
• PATH may omit /snap/bin → kubectl “not found”
• KUBECONFIG may point nowhere → connects to localhost:8080

Explicitly exporting them ensures kubectl finds your cluster and runs correctly.

Lesson learned: never assume environment variables exist in automation.

---

🩺 8. Troubleshooting Checklist

Add this quick debug command to your EC2 node:

kubectl -n ohwise get deploy ohwise-web-hub -o wide
kubectl -n ohwise get pods -l app=ohwise-web-hub -o=custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[*].image

---

🧭 9. Why This Approach Fits Ohwise

Ohwise builds distributed, AI-driven systems that require fast iteration and full transparency. We don’t want to hide critical infrastructure behind managed abstractions.

By running Kubernetes directly on EC2 and controlling deployments via SSM:

• We keep fine-grained control over versions, nodes, and cost.
• We stay cloud-agnostic — tomorrow this can run on any VM.
• We maintain a lightweight DevOps footprint — just Docker Hub, GitHub, and AWS IAM.

This setup scales linearly as we add more EC2 nodes; each simply needs the SSM Agent and a valid kubeconfig.

---

🧾 10. Future Enhancements

Once your base pipeline works, consider layering:

• Health checks & automatic rollback: detect unhealthy pods and revert the image.
• Slack notifications: post deployment results to your team chat.
• S3 or CloudWatch output: stream SSM command logs for permanent audit.
• Multi-cluster rollouts: loop through multiple SSM_INSTANCE_IDs.
• Blue-green deployments: use labels or ArgoCD for zero downtime.

This architecture is intentionally simple yet extensible — a strong base for more advanced DevOps practices.

---

If this guide helped you ship faster or understand the internals of CI/CD better: Reach out at https://heunify.com/contact if you want to collaborate or discuss system design, ML ops, or distributed architectures.