Skip to content

Commit

Permalink
Refactor to deployment
Browse files Browse the repository at this point in the history
  • Loading branch information
Patrick Easters committed Jun 13, 2022
1 parent e9bbf55 commit e864cbf
Show file tree
Hide file tree
Showing 9 changed files with 38 additions and 50 deletions.
4 changes: 2 additions & 2 deletions Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@ type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.1.0
version: 0.2.0

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
appVersion: 0.1.0
appVersion: 0.2.0
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,9 @@ A Helm chart to delete pods stuck in a pending state without image pull secrets

It is recommended to verify the service account is provisioned with secrets before creating deployments, but this chart is intended for cases where that is not feasible.

This chart is intended to be run after any needed application charts have been deployed to
an OpenShift cluster. The chart will create RBAC resources and deploy a job using the `openshift4/ose-cli` image. The job runs a Bash script that uses kubectl to identify pods stuck pending without image pull secrets, and deletes them. The script searches for pods multiple times to ensure deployment controllers have ample time to create pods.
This chart is intended to be run during the installation of other application charts on the cluster. The chart will create RBAC resources and a single-pod deployment using the `openshift4/ose-cli` image. The pod runs a Bash script that uses kubectl to identify pods stuck pending without image pull secrets, and deletes them. This script repeats at an interval configurable by the `waitIntervalSeconds` value.

This chart can be uninstalled once the application charts have been successfully installed.

# Deployment
This chart can be installed directly from a release archive or by cloning this repo locally.
Expand Down
21 changes: 11 additions & 10 deletions reconcile.sh
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
#!/bin/bash

wait=5
attempts=6
# wait interval (seconds) between checks
: "${WAIT_INTERVAL:=30}"

for i in $(seq 1 $attempts); do
while true; do
echo "Finding Pending pods with no image pull secrets..."
pods=$(oc get po --all-namespaces --field-selector=status.phase==Pending -o go-template='{{range .items}}{{.metadata.namespace}} {{.metadata.name}} {{ len .metadata.ownerReferences }} {{ len .spec.imagePullSecrets }}{{"\n"}}{{end}}')


count=0
# loop over all pending pods
while IFS= read -r line; do
if [ ! -z "$line" ]; then

ns="$(echo $line | cut -d ' ' -f 1)"
name="$(echo $line | cut -d ' ' -f 2)"
owners="$(echo $line | cut -d ' ' -f 3)"
Expand All @@ -19,14 +19,15 @@ for i in $(seq 1 $attempts); do
# if pod has an owner and no image pull secrets, delete it
if [ "$owners" -gt 0 ] && [ "$imgpull" -lt 1 ]; then
oc delete po -n $ns $name
((count++))
fi
fi
done <<< "$pods"

if [ "$i" -lt "$attempts" ]; then
echo "Checking again in $wait seconds"
sleep $wait
if [ "$count" -eq 0 ]; then
echo "No matching pods found"
fi
done

echo "Completed removing pods without image pull secrets"
# pause before checking again
sleep $WAIT_INTERVAL
done
34 changes: 17 additions & 17 deletions templates/job.yaml → templates/deployment.yaml
Original file line number Diff line number Diff line change
@@ -1,36 +1,36 @@
apiVersion: batch/v1
kind: Job
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "pod-pullsecret-reconciler.fullname" . }}
labels:
{{- include "pod-pullsecret-reconciler.labels" . | nindent 4 }}
annotations:
"helm.sh/hook": post-install
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
backoffLimit: 5
activeDeadlineSeconds: 1800
replicas: 1
selector:
matchLabels:
{{- include "pod-pullsecret-reconciler.selectorLabels" . | nindent 6 }}
template:
metadata:
{{- with .Values.podAnnotations }}
annotations:
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "pod-pullsecret-reconciler.selectorLabels" . | nindent 8 }}
spec:
serviceAccountName: {{ include "pod-pullsecret-reconciler.serviceAccountName" . }}
restartPolicy: Never
restartPolicy: Always
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
env:
- name: WAIT_INTERVAL
value: "{{ .Values.waitIntervalSeconds }}"
resources:
{{- toYaml .Values.resources | nindent 12 }}
command:
- /bin/bash
- /script/reconcile.sh
- /bin/bash
- /script/reconcile.sh
volumeMounts:
- name: script
mountPath: /script
- name: script
mountPath: /script
volumes:
- name: script
configMap:
Expand Down
7 changes: 2 additions & 5 deletions templates/role-binding.yaml
Original file line number Diff line number Diff line change
@@ -1,16 +1,13 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: pod-pullsecret-reconciler
name: {{ include "pod-pullsecret-reconciler.fullname" . }}
labels:
{{- include "pod-pullsecret-reconciler.labels" . | nindent 4 }}
annotations:
"helm.sh/hook": post-install
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded,hook-failed
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: pod-pullsecret-reconciler
name: {{ include "pod-pullsecret-reconciler.fullname" . }}
subjects:
- kind: ServiceAccount
name: {{ include "pod-pullsecret-reconciler.serviceAccountName" . }}
Expand Down
5 changes: 1 addition & 4 deletions templates/role.yaml
Original file line number Diff line number Diff line change
@@ -1,12 +1,9 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: pod-pullsecret-reconciler
name: {{ include "pod-pullsecret-reconciler.fullname" . }}
labels:
{{- include "pod-pullsecret-reconciler.labels" . | nindent 4 }}
annotations:
"helm.sh/hook": post-install
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded,hook-failed
rules:
- apiGroups:
- ""
Expand Down
3 changes: 0 additions & 3 deletions templates/script.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,6 @@ metadata:
name: pod-pullsecret-reconciler-script
labels:
{{- include "pod-pullsecret-reconciler.labels" . | nindent 4 }}
annotations:
"helm.sh/hook": post-install
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded,hook-failed
data:
reconcile.sh: |-
{{ .Files.Get "reconcile.sh" | indent 4 }}
3 changes: 0 additions & 3 deletions templates/serviceaccount.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,3 @@ metadata:
name: {{ include "pod-pullsecret-reconciler.serviceAccountName" . }}
labels:
{{- include "pod-pullsecret-reconciler.labels" . | nindent 4 }}
annotations:
"helm.sh/hook": post-install
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded,hook-failed
6 changes: 2 additions & 4 deletions values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ image:
pullPolicy: IfNotPresent
tag: v4.8

# number of seconds to wait between executions
waitIntervalSeconds: 30

nameOverride: ""
fullnameOverride: ""
Expand All @@ -16,10 +18,6 @@ serviceAccount:

podAnnotations: {}

service:
type: ClusterIP
port: 80

resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
Expand Down

0 comments on commit e864cbf

Please sign in to comment.