Fix Services in CrashLoop because of "Required migrations are outdated, waiting for their execution..."

How to re-run migrations job in Linux

Introduction

Starting in L8.2.4, the API-Gateway service runs a migration module as one of it's initialization containers (called check-containers).

In some cases, the migration does not run successfully and may cause NodeJS-based Pods to be stuck in CrashLoopBackOff. Taking the Identity Service as an example, the following was be printed in it's service logs:

sisense, identity, [INFO] [identity-service] [45]:[MigrationsRunner] [/usr/src/app/node_modules/@sisense/sisense-migrations/src/MigrationsRunner.js:135] [] [Required migrations are outdated, waiting for their execution...]
 
sisense, identity, [INFO] [identity-service] [45]:[] [/usr/src/app/node_modules/@sisense/sisense-logger/lib/init.js:168] [] [Service 'identity-service' exited with code '0'.]

This means that the Identity Service was waiting for the migration to complete and because it was not done, the service exited gracefully.

Solution

1) Take a backup of the migration job specification:

kubectl -n sisense get job migration -o yaml > job_migration.yaml

2) Rerun the job:

kubectl get job -n sisense migration -o json | jq 'del(.spec.selector)' | jq 'del(.spec.template.metadata.labels)' | kubectl replace --force -f -

3) Recycle service:

kubectl delete -n sisense pod -l app=identity

intapiuser

Posted 3 years ago

0 comments