Triaging Build Failures: "BE#967529 Failed to connect to new ec" and "BE#150935 Timeout"

[Oleg_S]

Triaging Build Failures

Symptoms

Sometimes during a build, you can encounter an error or two. Below are two common build failures you might experience and steps you can take to resolve the problem quickly.

"BE#967529 Failed to connect to new ec: <cube name>." This error happens during build initialization.

or you can receive

"BE#150935 Timeout" on a newly built cube which could happen during finalization.

Diagnosis

These errors mean that the Management service was not able to connect to the ec-bld pod which runs the build. There are a couple of reasons for this error, including: 

• management service tried 90 times to connect and was not able to
• initialization or finalization took more than 40 minutes
• error on the Kubernetes level

Solution

 First, we should understand it is OK for a cube to take longer to initialize. Common reasons include:

• large cube (100+GB)
• accumulative build
• slow server or disk

If you do not want to increase the specs of the deployment or increase disk performance, try increasing timeouts, which should result in a longer wait but a successful build.

Build connection retries

Check if the retries failed by looking for the words "retrying 1/90" (This is a countdown from 90/90 to 1/90) in the management pod logs:

kubectl -n <namespace> management-<press Tab to add full name> | grep 'retrying 1/90'

If you see a similar entry, you can update the Build Connection Retries:

1. Go to the Admin > System Management > Configuration

 

2. Scroll down to the Management section, expand it, update Build Connection Retries to 250 and Save changes. This will trigger the management pod to restart. 

Initialization can take more than 40 minutes.

If the initialization or finalization steps take more than 40 minutes, it could be due to similar reasons, slow disk or a large accumulative build.

Use the steps below to increase the default 40-minute value:

1. Go to the Admin > System Management > Configuration

 

2. Click 5 times on the Sisense logo and a hidden menu will appear.

3. Navigate to the Base Configuration menu, scroll to the ServiceCallTimeout value, and set it to 3600000 (which is 1 hour in milliseconds and can be set for more if need be). Click Save Base in the top right corner. Restart the Build, management pods, and run the build again.

Error on the Kubernetes level

If ec-bld pod or ec-qry pod can't start during the build process because of the Kubernetes level errors, services will not be able to connect, which will generate an error. For these case, please check the errors using kubectl. Below are some useful commands:

kubectl -n <namespace> describe pod <name of the pod>

kubectl -n <namespace> get events

kubectl -n <namespace> get nodes