All the indexing data lost after restarting container

[jkjha]

Describe the bug

I am using opengrok v1.12.12 in container. My indexer data is under a VM which is mounted to the container. I have 2 replicas of the opengrok pod that is running in my k8s environment.

The deployment was running fine, but we noticed that the newly added projects are sometimes not getting listed on the projects dropdown list. On further investigation we found that, when UI hits one of the replica it ruturns the old project list and so new projects are not visible, however, when it hits other replica, it returns the newly added projects as well.

To fix this, we tried to restart the deployment (Rollback and recreate the pod), but after recreating the pod, we were surprised to see data all the indexer data is gone. This was completely unexpected and our almost 3TB of indexed data was deleted.

Can you please check in priority what could have caused this and how to avoid it. And any thing we can do to avoid it?
This has caused too much impact on our productivity.

To Reproduce

1. Opengrok container with 2 replicas.
2. restart and rollback the deployment
3. Indexer data is lost.

(I have not tried above steps again)

Env variables:

        - env:
            - name: SYNC_PERIOD_MINUTES
              value: "120"
            - name: INDEXER_OPT
              value: "-A .vmodl:JavaAnalyzer"
            - name: WORKERS
              value: "10"
            - name: CHECK_INDEX
              value: "true"
            - name: NOMIRROR
              value: "1"
            - name: API_TIMEOUT
              value: "300"
            - name: OPENGROK_SCAN_REPOS
              value: "on"
            - name: OPENGROK_SCAN_DEPTH
              value: "20"
            - name: OPENGROK_GENERATE_HISTORY
              value: "on"
            - name: OPENGROK_SUGGESTTER_ENABLED
              value: "false"
            - name: P4CONFIG
              value: ".p4config"
            - name: P4USER
              value: "censored"
            - name: P4TICKETS
              value: "/opengrok/data/secrets/p4tickets"
            - name: P4TRUST
              value: "/opengrok/data/secrets/p4trust/.p4trust"
            - name: READONLY_CONFIG_FILE
              value: "/opengrok/etc/read-only.xml"

[jkjha]

Request to look into this with priority.

[jkjha]

I see this in the log... what made this to wipe out all the data? Even though the previous version was exact same...

Index check failed
Aug 16, 202307:33:40.000
You might want to remove all data under the data root and reindex
Aug 16, 202307:33:40.000
Wiping out data root

[jkjha]

Hi Opengrok team, any update on this. We started reindexing whole projects again which is still ongoing (900 out of 1200 are done) for >3TB of projects. We need your help to avoid this in future.

We have turned off the CHECK_INDEX flag, which we suspect might be the cause of the problem. However, we still require assistance to verify whether this is indeed the issue or if there's something else at play.

[vladak]

As for why the index check failed, this is normally logged with the WARNING log level:

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/index/IndexCheck.java

Line 178 in f412b9b

LOGGER.log(Level.WARNING, String.format("Index check for directory '%s' failed", indexPath), e);

so I believe should be part of the container logs since the indexer command is run with doprint=True in start.py.

[jkjha]

Thanks for looking into this...I will try to find more logs from container to check there are any more info...meanwhile, please help with few queries...

under what case index check could fail? There was no change in the image, it was just a rollback restart of the pod (running 2 pods). My assumption was if there are major change in docker image version and the new upgraded image is not compatible with previous index data, then only it will delete.

Also if index check fails for any one particular repository, will it wipe out whole data folder?

I have another issue while running multiple pod where one of them are not in sync with updated project list... but that we can discuss later in separate thread.

[vladak]

The index check has several modes of operation. The basic mode used by the Docker image merely checks the version stored in the segments in the index/indices. If the version embedded into the running indexer does not match the version stored in the index, the indexer will report it and start.py performs the index deletion. Looking at the code

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/index/IndexCheck.java

Lines 174 to 180 in f412b9b

	try {
	LOGGER.log(Level.INFO, "Checking index in ''{0}''", indexPath);
	checkDir(indexPath, mode, projectName);
	} catch (Exception e) {
	LOGGER.log(Level.WARNING, String.format("Index check for directory '%s' failed", indexPath), e);
	return 1;
	}

the Exception is maybe too broad as it can hide IOException which is the case where the version check (or the more thorough document check) could not be performed and it should just gracefully exit. This could be what happened in your deployment - possibly for some reason the index files were not accessible during the index check.

[vladak]

The index check code will receive the IOException treatment via PR #4402.

[jkjha]

Thanks for looking into this.