Global Search deployment and operations runbook

Neutrinos Platform Product Release Notes
,
1

Applies to Alpha Trinity asset 26.10.1.3.0.

Global Search is a preview feature, disabled by default. Follow this runbook
to opt into the preview.

Global Search indexes case and task data from the enquiry database. Case service
applies the search queue migrations at startup when GLOBAL_SEARCH is enabled.
The index worker processes queued changes. Existing cases require a backfill.

Components

Component Role
Meilisearch HA cluster External search engine, deployed through the Marketplace.
alpha-global-search-service Search API, streamed results and search backfill API.
alpha-global-search-index-worker Uses the search service image in index-worker mode. Reads the enquiry queue and writes to Meilisearch.
alpha-case-service Applies queue migrations, queues enquiry changes and runs permission-filtered searches with the search-only key.
gts Queues task projection changes in the same transaction as the projection write.
alpha-case-manager-ui Enables the search box when both required flags are on.

Deployment sequence

  1. Complete the prerequisites in section 1.
  2. Deploy Meilisearch HA and create API keys in Step A.
  3. Configure the Alpha values in Step B.
  4. Choose startup migration or the optional migration Job in Step C.
  5. Deploy with indexing paused in Step D.
  6. Resume indexing in Step E.
  7. Backfill each project in Step F. Skip this for an environment with no cases.
  8. Verify the deployment in Step G.

To re-enable an existing deployment, follow section 9.2.

1. Access and prerequisites

Required access:

  • Trinity asset values or Helm, and kubectl access to the target namespace.
  • An Alpha browser session for search service API requests.
  • A SQL client connected to the enquiry database.

Replace angle-bracket placeholders with the target environment’s values.

1.1 Service paths

Service Base URL
Search API <domain>/global-search
Case service <domain>/caseservice
Index worker In-cluster only, port 8080, /global-search

With ENABLE_SWAGGER=true, open <domain>/global-search/api-docs and select
“Search index backfill”. All backfill operations use POST.

1.2 Authentication

The search service requires a browser session. It rejects IDS client-credentials
tokens and service JWTs.

  1. Log in to Case Manager as an administrator.
  2. Copy the session cookie from browser developer tools. Its name comes from
    ALPHA_SESSION_NAME, usually alpha.sessionid.
  3. Request a CSRF token:
GET <domain>/global-search/csrf/token
Cookie: alpha.sessionid=<cookie>

The response contains csrfToken. Send the cookie and x-csrf-token: <token> on
each POST. A missing or expired session returns 401. Swagger on the same host
uses the browser session, but POST requests still require the CSRF token.

Use the authentication procedure in the inbox sorting runbook for its backfill
APIs. GET /caseservice/ping-meili is public.

1.3 Databases

The enquiry database holds the read model, search queue and backfill progress.
The search service and worker inherit ENQUIRY_DB_NAME and
ENQUIRY_DB_SCHEMA_NAME from services.backends.alpha-case-service.envs, and the
connection from features.database. The default schema is enquiry_schema.
The worker does not need Case Manager, BPM or session database access.

1.4 Global Search requires CASE_FETCH_ALL_PERF_OPTIMIZATION

Enable CASE_FETCH_ALL_PERF_OPTIMIZATION in features.featureFlags and retain it
on case service, JBPM service, gts and Case Manager. It keeps the enquiry read
model current when BPM changes tasks through timers, escalations or reassignments.
The chart does not enforce this dependency. Without it, Case Manager displays a
disabled search box and links to the prerequisite runbook.

Complete the asset 1.0.17+ migrations and each project’s BPM lineage and enquiry
projection backfills. Follow the
server-side sorting migration runbook
or docs/migration-apis-doc.md.

Search backfill requires populated ci_* and ti_* enquiry tables. Empty
projections provide no historical data; missing tables can fail the backfill.

1.5 Projects to index

Every project with an enquiry configuration needs one search backfill. List them:

SELECT "projectId", "caseType", "caseDataTableName", "taskDataTableName"
FROM <enquiry-schema>.enquiry_config
ORDER BY "projectId";

Record each projectId and confirm that both table names are present. Run Step F
for each project. Count its cases to estimate the backfill size:

SELECT COUNT(*) FROM <enquiry-schema>.<ci_*>;   -- cases to index for this project

2. Step A. Deploy the Meilisearch HA cluster

Deploy the managed Meilisearch HA cluster separately from Alpha. It uses engine
pods, a Redis Streams write-ahead log and sidecars. The read load balancer routes
to engines that have caught up and accepts both reads and writes. Keep
MEILISEARCH_MODE=ha, the default; the Alpha client rejects a plain single-node
Meilisearch deployment.

2.1 Install from the Marketplace

  1. In Trinity or the Marketplace, open the Meilisearch HA asset approved for this
    release. Deploy it to the cluster, normally into its own meilisearch
    namespace. One HA cluster can serve several Alpha environments, because each
    environment uses its own index prefix. See section 3.2.
  2. Set the master key in the asset values. Use at least 32 characters from a
    secret store, and never reuse a sample value.
  3. Wait until every engine pod and the read load balancer are ready:
kubectl -n meilisearch get pods
kubectl -n meilisearch get svc | grep read-lb
  1. Record the read load balancer address. In the cluster it is normally:
http://meilisearch-read-lb.meilisearch.svc.cluster.local:7700
  1. Check its health. This call needs no key:
kubectl -n <alpha-namespace> run meili-check --rm -it --image=curlimages/curl --restart=Never -- \
  curl -s http://meilisearch-read-lb.meilisearch.svc.cluster.local:7700/health
# {"status":"available"}

Use the Meilisearch asset’s deployment guide for replicas, storage and Redis
configuration.

2.2 Create two API keys

Create separate write and search-only keys. Assign only the search-only key to
case service.

Key Goes to Actions Indexes
Write key alpha-global-search-index-worker documents.add, documents.delete, indexes.create, indexes.get, indexes.delete, settings.update <indexPrefix>-cm-runtime-*
Search-only key alpha-case-service search <indexPrefix>-cm-runtime-*

Create both with the master key through the standard Meilisearch keys API, or
follow the platform team’s procedure for the HA cluster:

MEILI=http://meilisearch-read-lb.meilisearch.svc.cluster.local:7700
PREFIX='<indexPrefix>'   # see §3.2

curl -s -X POST "$MEILI/keys" -H "Authorization: Bearer <master-key>" -H 'Content-Type: application/json' -d "{
  \"description\": \"alpha $PREFIX index worker\",
  \"actions\": [\"documents.add\",\"documents.delete\",\"indexes.create\",\"indexes.get\",\"indexes.delete\",\"settings.update\"],
  \"indexes\": [\"$PREFIX-cm-runtime-*\"], \"expiresAt\": null }" | jq -r .key

curl -s -X POST "$MEILI/keys" -H "Authorization: Bearer <master-key>" -H 'Content-Type: application/json' -d "{
  \"description\": \"alpha $PREFIX case-service search\",
  \"actions\": [\"search\"],
  \"indexes\": [\"$PREFIX-cm-runtime-*\"], \"expiresAt\": null }" | jq -r .key

2.3 Store the keys as Kubernetes Secrets

Create both Secrets in the Alpha namespace:

kubectl -n <alpha-namespace> create secret generic alpha-search-write-key --from-literal=api-key='<write-key>'
kubectl -n <alpha-namespace> create secret generic alpha-search-read-key  --from-literal=api-key='<search-only-key>'

3. Step B. Prepare the Alpha values

Configure the environment’s Trinity asset values or Helm values file. Supply
values explicitly when running Helm; the chart has no values.yaml.

3.1 Feature flags

To enable the preview, add GLOBAL_SEARCH to the environment’s complete features.featureFlags list.
An explicit list replaces the defaults. Preserve existing flags. For example:

features:
    featureFlags:
        - ADMIN_FILTERS
        - PULL_TASK
        - INBOX_PROJECTION
        - GLOBAL_CUSTOM_CODE
        - BPM_CONFIG_VERSION
        - CHANGE_TRACKING
        - FALLBACK_TO_MAIN_CASEDEF
        - CASE_FETCH_ALL_PERF_OPTIMIZATION # required by GLOBAL_SEARCH
        - CASE_DATA_VERSIONING
        - SNAPSHOT_CONTENT_JSONB
        - GLOBAL_SEARCH # enable in Step D

The chart renders FEATURE_FLAGS as a JSON array. Services read it at startup.
A per-service envs.FEATURE_FLAGS override must be a literal JSON array string.
It can disable GLOBAL_SEARCH for that service but cannot enable it when the
deployment-wide list omits it. Keep CASE_FETCH_ALL_PERF_OPTIMIZATION as required
by section 1.4.

3.2 Search engine settings

features:
    search:
        enabled: true
        url: http://meilisearch-read-lb.meilisearch.svc.cluster.local:7700
        indexPrefix: <customer>-<env> # for example acme-prod
        apiKey:
            valueFrom:
                secretKeyRef:
                    name: alpha-search-write-key
                    key: api-key
        readApiKey:
            valueFrom:
                secretKeyRef:
                    name: alpha-search-read-key
                    key: api-key

url and indexPrefix become MEILISEARCH_HOST and MEILISEARCH_INDEX_PREFIX
on case service and the worker. Each receives its own MEILISEARCH_API_KEY:
apiKey for worker writes and readApiKey for case-service searches. Keys accept
a plain string or valueFrom.

Use a unique, stable prefix per environment. It may contain letters, digits,
hyphens and underscores, up to 100 characters. Index names are
<indexPrefix>-cm-runtime-<projectId>.

With search enabled, the chart requires the URL, prefix and both keys.
enabled: false removes the worker and case-service Meilisearch settings but
keeps the API and producers. Use the worker pause setting in section 9.1 to pause
indexing. Remove obsolete features.search.masterKey and meilisearch: settings.

Do not change the URL or prefix while the queue has pending work. After changing
either, repeat Step F for every project.

3.3 Enquiry database and worker database role

Set the enquiry database and schema on case service. The chart uses these values
for the search service and worker:

services:
    backends:
        alpha-case-service:
            envs:
                ENQUIRY_DB_NAME: <enquiry-db>
                ENQUIRY_DB_SCHEMA_NAME: <enquiry-schema>

To use a restricted worker role instead of the shared credentials in
features.database, set these overrides:

services:
    backends:
        alpha-global-search-index-worker:
            envs:
                DB_USERNAME: <search_worker_role>
                DB_PASSWORD:
                    valueFrom:
                        secretKeyRef: { name: alpha-search-worker-db, key: password }

Apply these grants after the queue tables exist. If Step D creates the tables,
apply the grants before resuming indexing in Step E. The search service’s database
role needs the same access for the backfill API. Grant access to new projection
tables when projects are added.

GRANT USAGE ON SCHEMA <enquiry-schema> TO <role>;
GRANT SELECT ON <enquiry-schema>.enquiry_config, <enquiry-schema>.projection_data TO <role>;
GRANT SELECT ON ALL TABLES IN SCHEMA <enquiry-schema> TO <role>;                 -- ci_* / ti_* projections
GRANT SELECT, INSERT, UPDATE, DELETE ON <enquiry-schema>.search_index_jobs, <enquiry-schema>.search_index_slots,
      <enquiry-schema>.search_index_blocks, <enquiry-schema>.search_index_backfills TO <role>;
GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA <enquiry-schema> TO <role>;

3.4 Service-to-service signing, session and ingress

  • Configure global.serviceAuth.internal.privateKey, publicKey and keyId.
    The worker requires the shared signing credentials and rejects service overrides.
  • The search service uses the session, IDS and database settings in
    features.session, features.ids and features.database.
  • Keep ingress.streaming: true and readTimeoutSeconds: 120. Disable buffering
    and compression for /global-search/* at every gateway. Allow at least 120
    seconds of read and idle time.
  • The chart sets ALPHA_UI_GLOBAL_SEARCH_SERVICE_URL to
    https://<host>/global-search. Override it only for a different route.

3.5 Optional search service overrides

Set overrides under services.backends.<service>:

Service Setting Default
alpha-global-search-service version global.imageTag
alpha-global-search-service envs.GSS_SEARCH_TIMEOUT_MS 5000
alpha-global-search-service envs.GSS_SUGGEST_TIMEOUT_MS 1500
alpha-global-search-service envs.GSS_SSE_HEARTBEAT_MS 15000
alpha-global-search-index-worker version API version, then global.imageTag

Remove old providers: and GLOBAL_SEARCH_PROVIDERS overrides. The service
registers Inbox when GLOBAL_SEARCH is enabled.

4. Step C. Choose how to apply the queue migrations

Case service applies missing queue migrations through version 4 at startup when
GLOBAL_SEARCH is enabled, before listening for requests. This covers new and
existing enquiry schemas. These migrations are separate from
migration-service-v2 and do not modify Meilisearch.

If the case-service database role can create and alter queue tables and indexes,
continue to Step D and verify section 4.3 afterward. Otherwise, use the Job in
section 4.2 with a suitable role before rollout.

The Job also avoids a startup race with gts, which checks for migration 2 but
does not apply it. If gts starts first, let it restart after migration completes.

Version Creates Needed by
1 search_index_jobs, search_index_slots, search_index_blocks, search_index_migrations worker
2 Indexes for claim order, retention and metrics case service and gts
3 search_index_backfills for backfill progress backfill API
4 Descending-scan columns on search_index_backfills backfill API in this release

4.1 Baseline

SELECT version, applied_at
FROM <enquiry-schema>.search_index_migrations ORDER BY version;
-- "relation does not exist" means no migration has run yet, expected on first install

4.2 Optional migration Job

Use the search service image tag for this release and the target enquiry database.
The database role must be able to create and alter the queue tables and indexes.
Save the following manifest as gss-migrate-index-queue.yaml:

apiVersion: batch/v1
kind: Job
metadata:
    name: gss-migrate-index-queue
spec:
    backoffLimit: 0
    template:
        spec:
            restartPolicy: Never
            imagePullSecrets: [{ name: <imagePullSecret> }]
            containers:
                - name: migrate
                  image: <registry>/<repository>/alpha-global-search-service:<tag>
                  command: ['node', 'main.js', '--migrate-index-queue']
                  env:
                      - { name: DB_HOST, value: '<db-host>' }
                      - { name: DB_PORT, value: '5432' }
                      - { name: DB_USERNAME, value: '<migration-role>' }
                      - name: DB_PASSWORD
                        valueFrom: { secretKeyRef: { name: <db-secret>, key: password } }
                      - { name: ENQUIRY_DB_NAME, value: '<enquiry-db>' }
                      - { name: ENQUIRY_DB_SCHEMA_NAME, value: '<enquiry-schema>' }
                      - { name: DB_SSL, value: 'false' } # plus DB_CA_CERT / DB_CA_CERT_PATH if SSL

kubectl -n <alpha-namespace> apply -f gss-migrate-index-queue.yaml
kubectl -n <alpha-namespace> wait --for=condition=complete job/gss-migrate-index-queue --timeout=10m
kubectl -n <alpha-namespace> logs job/gss-migrate-index-queue
kubectl -n <alpha-namespace> delete job gss-migrate-index-queue

The migration takes a PostgreSQL advisory lock. Concurrent migration runs wait
for that lock. Migration 2 locks tables while it builds indexes. Schedule a
maintenance window if an existing large queue still needs that migration.

4.3 Verify

Run these queries after case-service startup or the migration Job completes:

SELECT version FROM <enquiry-schema>.search_index_migrations ORDER BY version;
-- expect 1, 2, 3, 4
SELECT to_regclass('<enquiry-schema>.search_index_backfills') IS NOT NULL AS backfill_table;  -- t

5. Step D. Deploy with indexing paused

Enable GLOBAL_SEARCH in the complete flag list from section 3.1. Pause the
worker with the override below. Producers continue adding changes to the queue
while the worker is paused.

services:
    backends:
        alpha-global-search-index-worker:
            envs:
                GSS_INDEX_WORKER_ENABLED: 'false'

Apply through Trinity, or run helm upgrade with the customer values file last,
and wait for the rollout:

kubectl -n <alpha-namespace> rollout status deploy/alpha-global-search-service
kubectl -n <alpha-namespace> rollout status deploy/alpha-global-search-index-worker
kubectl -n <alpha-namespace> rollout status deploy/alpha-case-service
kubectl -n <alpha-namespace> rollout status deploy/gts
kubectl -n <alpha-namespace> rollout status deploy/alpha-case-manager-ui

Checks:

  1. Check the rendered flags in the case-service, gts and Case Manager pods.
    For example:
kubectl -n <alpha-namespace> exec deploy/alpha-case-service -- printenv FEATURE_FLAGS
# must contain both "CASE_FETCH_ALL_PERF_OPTIMIZATION" and "GLOBAL_SEARCH"
  1. GET <domain>/global-search/alive is public and returns
    {"status":"ok","service":"alpha-global-search-service"}.
  2. The search service startup log lists the providers:
    1 of 1 provider(s) registered: inbox.
  3. GET <domain>/caseservice/ping-meili returns meilisearch: "ok".
    meilisearch: "unavailable" indicates a URL, key or network problem.
    meilisearch: "disabled" means case service has not enabled the search client.
  4. The worker is up but paused. kubectl logs deploy/alpha-global-search-index-worker
    shows no claims, and /global-search/ready returns 200 inside the cluster.
  5. Verify queue migrations 1 through 4 using section 4.3. Apply any database grants
    from section 3.3 before resuming indexing.

6. Step E. Resume indexing

  1. Set GSS_INDEX_WORKER_ENABLED to "true", or remove the override to use that
    default:
services:
    backends:
        alpha-global-search-index-worker:
            envs:
                GSS_INDEX_WORKER_ENABLED: 'true'
  1. Apply, then wait for rollout status deploy/alpha-global-search-index-worker.
  2. Hard-refresh Case Manager. The search box appears in the header. It finds only
    what the worker has indexed so far. Step F indexes the history.

Every service reads the flag once at startup. A change to features.featureFlags
takes effect only after Helm restarts the pods. Never edit FEATURE_FLAGS inside a
running pod.

7. Step F. Backfill each project

The backfill indexes one project’s existing enquiry rows, the ci_* cases and
their ti_* tasks, through the queue. It never reads Case Manager or BPM. It
queues one case, waits until Meilisearch confirms the case is visible, then moves
the cursor. Requests process cases sequentially, regardless of batchSize.

Use the base URL <domain>/global-search/migration/search-index-backfill for the
POST endpoints below. Include the session cookie and x-csrf-token from section
1.2.

7.1 When to run it

Situation Action
First deployment with existing cases Run up for every project.
Search was disabled, the engine was replaced or emptied, or the prefix changed Run up with resetCursor: true for every project.
A project’s index needs rebuilding Run down, then up. See section 9.3.
A new project or enquiry projection repair No separate search backfill. Writes and repairs queue their changes while search is enabled.
Searchable fields changed The worker updates settings. Run up with resetCursor: true to rebuild existing documents immediately.

7.2 Read the baseline with POST /status/project

{ "projectId": "<project-uuid>" }

Check caseType, totalCases, remainingCases and job. A project with no
saved backfill has job: null. If the API reports missing enquiry configuration,
complete the projection backfill in section 1.4.

7.3 Preview with POST /up/project

{ "projectId": "<project-uuid>", "dryRun": true }

The response has the same counts plus hasMore, order and dryRun: true. A
dry run queues nothing and leaves progress unchanged.

7.4 Run the backfill with POST /up/project

Start a new scan with one case:

{ "projectId": "<project-uuid>", "resetCursor": true, "batchSize": 1, "maxBatches": 1 }

A completed one-case request returns processedThisRun: 1. It can return
202 pending if indexing is still in progress. Continue with the request below,
without resetCursor, until hasMore is false:

{ "projectId": "<project-uuid>", "batchSize": 100, "maxBatches": 20, "batchDelayMs": 200 }

Check hasMore after each call. processedThisRun counts confirmed cases and
durationMs reports elapsed time.

A request waits up to 30 seconds for indexing across its batches. HTTP 202 with
status: "pending" and hasMore: true means indexing is incomplete. Retry without
resetCursor, including after a timeout or disconnect. The service saves progress
and reuses an unexpired pending ticket. It can enqueue the case again after the
ticket expires.

Request fields:

Field Default Range Meaning
projectId none required Project to index
order desc asc or desc Case ID order. desc indexes the newest cases first
batchSize 100 1 to 1000 Cases read per batch
maxBatches 20 1 to 100 Batches per request. The 30-second limit usually ends the request first
batchDelayMs 200 0 to 5000 Pause between batches, in milliseconds
resetCursor false true or false Start a new scan from the top. Use it once at the start, never on a retry
dryRun false true or false Preview only

A retry without order keeps the saved order. To change the order, send
resetCursor: true, or the service returns 409. A descending scan records the
highest case ID when it starts. Normal writes index cases created after that, so
the scan never sees them. To scan again after a scan completes, send
resetCursor: true.

7.5 Verify with POST /status/project

Completion requires remainingCases: 0 and job.status: "completed".

job.status is one of idle, running, pending, completed or failed.
Retry a pending request. For a failed request, inspect lastError. Verify the
saved progress in SQL:

SELECT project_id, operation, status, cases_processed, sort_order, cursor_cid, snapshot_max_cid, last_error, updated_at
FROM <enquiry-schema>.search_index_backfills WHERE project_id = '<project-uuid>';

Then search Case Manager for the ID of an old case and for a field value from an
old case.

7.6 Several projects

Run only one backfill request per project at a time. Concurrent requests can
return 409 Backfill progress changed in another request. Different projects
share the queue’s indexing slots, which default to four. Check worker capacity
and live indexing load before increasing concurrency.

8. Step G. Verify the deployment

  1. Run the smoke script from an alpha-core checkout:
node scripts/smoke/global-search-smoke.mjs --url https://<host>/global-search
node scripts/smoke/global-search-smoke.mjs --url https://<host>/global-search --cookie 'alpha.sessionid=<cookie>' --project <projectId>

The script checks /alive, the 401 answers without a session, that /configs
lists inbox, and that the stream is uncompressed text/event-stream in several
chunks. If results arrive only at the end, check gateway buffering and compression
as described in section 3.4.

  1. Check that no queue job is dead or stuck:
SELECT status, COUNT(*) FROM <enquiry-schema>.search_index_jobs GROUP BY status;
-- pending and processing should fall to about 0; dead should be 0

SELECT id, project_id, cid, kind, attempts, last_error, updated_at
FROM <enquiry-schema>.search_index_jobs WHERE status = 'dead' ORDER BY updated_at DESC LIMIT 20;

SELECT * FROM <enquiry-schema>.search_index_blocks;   -- projects or branches blocked by missing config
  1. Scrape the worker’s internal /global-search/metrics. It reports queue depth
    and age, active slots, the largest indexing lag over the last 5 minutes,
    failures, and writes the cluster accepted but has not yet confirmed.
  2. Claim or reassign a task in Case Manager, then confirm search finds the change
    within a few seconds.
  3. In the UI, the search box shows “Case Manager” results with My Tasks, Group
    Tasks, Team Tasks and Enquiry sections, filtered to the user’s permissions.

A healthy /alive or a successful Helm rollout does not prove that the history is
indexed.

9. Operations, rollback and troubleshooting

9.1 Pause and resume indexing

To pause without losing work, keep GLOBAL_SEARCH on and set
alpha-global-search-index-worker.envs.GSS_INDEX_WORKER_ENABLED: "false". Producers
keep adding to the queue, the worker claims nothing new, active jobs finish and
cleanup still runs. Backfill requests return 202 pending while indexing is
paused. Set the value to "true" to resume.

9.2 Turn search off and on again

To turn search off, remove GLOBAL_SEARCH from features.featureFlags and deploy.
The chart removes the API and the worker, Case Manager hides the box, and the
producers stop adding to the queue, so nothing records the changes made while
search is off. Keep the queue tables and the Meilisearch indexes.

To re-enable search, restore the flag and follow Steps D and E. Then run Step F with
resetCursor: true for every project to index the changes made while it was off.

9.3 Rebuild a project index with POST /down/project

{ "projectId": "<project-uuid>" }

This call deletes only the project’s Meilisearch index and its search backfill
progress. It leaves ci_*, ti_*, si_* and the source data alone. On
202 pending, retry until the answer is
"Search index deleted; enquiry projections are unchanged". Then run up as described in section 7.4.
While a deletion is pending, up returns 409. A write after the delete can
create the index again right away.

Do not use the case-service .../enquiry-projection-backfill/down/project for
search. It truncates the enquiry tables and leaves the search index alone.

9.4 Dead jobs and capacity

Run these from an alpha-core checkout with the enquiry DB environment variables
set:

node --import tsx scripts/index-queue.ts replay <JOB_ID>     # reopen a dead job, then reconcile
node --import tsx scripts/index-queue.ts capacity <N>        # global indexing slots, 1 to 64; default 4; drain before shrinking
node --import tsx scripts/index-queue.ts migrate             # same as the migration Job

The worker marks a job dead after its fifth failed attempt. The wait between
attempts starts at 30 seconds and grows to at most 5 minutes. Fix the cause, such
as the key, the network or the configuration, before replaying the job.

9.5 Roll back the application

Before reverting images, check the target release’s rollback requirements. Keep
the queue schema and Meilisearch indexes. Do not drop the search_index_* tables
or resume a descending
backfill with an image older than this release, because older images only support
ascending scans.

9.6 Troubleshooting

Symptom Check or action
No API or worker Deployment Check global and service flags. For the worker, check features.search.enabled.
Helm reports missing Meilisearch or enquiry settings Complete sections 3.2 and 3.3. Keys must be strings or valueFrom.secretKeyRef maps.
Case service or gts reports missing migration 2 Check case-service migration logs and database permissions. Verify section 4.3, then restart. Use the optional Job if needed.
Search box disabled Complete section 1.4 and restore CASE_FETCH_ALL_PERF_OPTIMIZATION.
Search box missing Check Case Manager’s image and FEATURE_FLAGS, then hard-refresh.
Results arrive together or search times out Disable gateway buffering and compression. Use a read timeout of at least 120 seconds.
Meilisearch unavailable or INDEX_UNAVAILABLE Check the URL, search-only key and network policies.
Worker reports UNAUTHORIZED Check write-key actions and the <indexPrefix>-cm-runtime-* index pattern.
Empty results without an error Check whether the project has an index. A missing index returns no results. Run Step F.
Backfill returns 400 for missing enquiry configuration Save the inbox configuration or complete the projection backfill.
Backfill returns 409 Finish concurrent requests or pending deletion. Changing scan order requires resetCursor.
Backfill stays pending Check worker pause settings, logs and dead jobs.
Backfill returns 503 Check GLOBAL_SEARCH and ENQUIRY_DB_NAME.
Backfill returns 401 Log in again and obtain a new session cookie and CSRF token.
search_index_blocks contains rows Restore the missing project or branch inbox configuration. The worker checks again each minute.

10. API reference

Search service base URL: <domain>/global-search. Use a session cookie and CSRF
token for authenticated POST requests.

Method Path Body
GET /alive None. Public.
GET /configs None.
POST /search, /suggest { q, projectId, providers? }. Returns SSE.
POST /migration/search-index-backfill/status/project { projectId }
POST /migration/search-index-backfill/up/project Fields in section 7.4.
POST /migration/search-index-backfill/down/project { projectId }

Worker probes and metrics use GET /alive, /ready and /metrics at
:8080/global-search inside the cluster. Case-service health uses public
GET /caseservice/ping-meili.

🔎 Alpha Platform Trinity Asset (1.3.0) release notes - 26.10.1.3.0