# Global Search deployment and operations runbook
**URL:** https://community.neutrinos.com/t/global-search-deployment-and-operations-runbook/2893
**Category:** Product Release Notes
**Tags:** neutrinos-alpha, alpha-release-notes
**Created:** [October 30, 2026, 2:18pm UTC](https://community.neutrinos.com/t/global-search-deployment-and-operations-runbook/2893 "2026-10-30T14:18:46Z")
**Posts on this page:** 1
**Page:** 1
### Author:  [@doomedCode](https://community.neutrinos.com/u/doomedCode)
#### Post date: [October 30, 2026, 2:18pm UTC](https://community.neutrinos.com/t/global-search-deployment-and-operations-runbook/2893/1 "2026-10-30T14:18:46Z")
> Applies to Alpha Trinity asset 26.10.1.3.0.
>
> Global Search is a **preview feature** , disabled by default. Follow this runbook
> to opt into the preview.
Global Search indexes case and task data from the enquiry database. Case service
applies the search queue migrations at startup when `GLOBAL_SEARCH` is enabled.
The index worker processes queued changes. Existing cases require a backfill.
## Components
| Component | Role |
| --- | --- |
| Meilisearch HA cluster | External search engine, deployed through the Marketplace. |
| `alpha-global-search-service` | Search API, streamed results and search backfill API. |
| `alpha-global-search-index-worker` | Uses the search service image in `index-worker` mode. Reads the enquiry queue and writes to Meilisearch. |
| `alpha-case-service` | Applies queue migrations, queues enquiry changes and runs permission-filtered searches with the search-only key. |
| `gts` | Queues task projection changes in the same transaction as the projection write. |
| `alpha-case-manager-ui` | Enables the search box when both required flags are on. |
## Deployment sequence
1. Complete the prerequisites in section 1.
2. Deploy Meilisearch HA and create API keys in Step A.
3. Configure the Alpha values in Step B.
4. Choose startup migration or the optional migration Job in Step C.
5. Deploy with indexing paused in Step D.
6. Resume indexing in Step E.
7. Backfill each project in Step F. Skip this for an environment with no cases.
8. Verify the deployment in Step G.
To re-enable an existing deployment, follow [section 9.2](#92-turn-search-off-and-on-again).
* * *
## 1. Access and prerequisites
Required access:
- Trinity asset values or Helm, and `kubectl` access to the target namespace.
- An Alpha browser session for search service API requests.
- A SQL client connected to the enquiry database.
Replace angle-bracket placeholders with the target environment’s values.
### 1.1 Service paths
| Service | Base URL |
| --- | --- |
| Search API | `/global-search` |
| Case service | `/caseservice` |
| Index worker | In-cluster only, port 8080, `/global-search` |
With `ENABLE_SWAGGER=true`, open `/global-search/api-docs` and select
“Search index backfill”. All backfill operations use `POST`.
### 1.2 Authentication
The search service requires a browser session. It rejects IDS client-credentials
tokens and service JWTs.
1. Log in to Case Manager as an administrator.
2. Copy the session cookie from browser developer tools. Its name comes from
`ALPHA_SESSION_NAME`, usually `alpha.sessionid`.
3. Request a CSRF token:
```http
GET /global-search/csrf/token
Cookie: alpha.sessionid=
```
The response contains `csrfToken`. Send the cookie and `x-csrf-token: ` on
each `POST`. A missing or expired session returns `401`. Swagger on the same host
uses the browser session, but `POST` requests still require the CSRF token.
Use the authentication procedure in the inbox sorting runbook for its backfill
APIs. `GET /caseservice/ping-meili` is public.
### 1.3 Databases
The enquiry database holds the read model, search queue and backfill progress.
The search service and worker inherit `ENQUIRY_DB_NAME` and
`ENQUIRY_DB_SCHEMA_NAME` from `services.backends.alpha-case-service.envs`, and the
connection from `features.database`. The default schema is `enquiry_schema`.
The worker does not need Case Manager, BPM or session database access.
### 1.4 Global Search requires `CASE_FETCH_ALL_PERF_OPTIMIZATION`
Enable `CASE_FETCH_ALL_PERF_OPTIMIZATION` in `features.featureFlags` and retain it
on case service, JBPM service, `gts` and Case Manager. It keeps the enquiry read
model current when BPM changes tasks through timers, escalations or reassignments.
The chart does not enforce this dependency. Without it, Case Manager displays a
disabled search box and links to the prerequisite runbook.
Complete the asset 1.0.17+ migrations and each project’s BPM lineage and enquiry
projection backfills. Follow the
[server-side sorting migration runbook](https://community.neutrinos.com/t/migration-backfill-for-server-side-sorting-in-alpha-operations-runbook-for-alpha-trinity-asset-v1-0-17/2133)
or `docs/migration-apis-doc.md`.
Search backfill requires populated `ci_*` and `ti_*` enquiry tables. Empty
projections provide no historical data; missing tables can fail the backfill.
### 1.5 Projects to index
Every project with an enquiry configuration needs one search backfill. List them:
```sql
SELECT "projectId", "caseType", "caseDataTableName", "taskDataTableName"
FROM .enquiry_config
ORDER BY "projectId";
```
Record each `projectId` and confirm that both table names are present. Run Step F
for each project. Count its cases to estimate the backfill size:
```sql
SELECT COUNT(*) FROM .; -- cases to index for this project
```
* * *
## 2. Step A. Deploy the Meilisearch HA cluster
Deploy the managed Meilisearch HA cluster separately from Alpha. It uses engine
pods, a Redis Streams write-ahead log and sidecars. The read load balancer routes
to engines that have caught up and accepts both reads and writes. Keep
`MEILISEARCH_MODE=ha`, the default; the Alpha client rejects a plain single-node
Meilisearch deployment.
### 2.1 Install from the Marketplace
1. In Trinity or the Marketplace, open the Meilisearch HA asset approved for this
release. Deploy it to the cluster, normally into its own `meilisearch`
namespace. One HA cluster can serve several Alpha environments, because each
environment uses its own index prefix. See section 3.2.
2. Set the master key in the asset values. Use at least 32 characters from a
secret store, and never reuse a sample value.
3. Wait until every engine pod and the read load balancer are ready:
```bash
kubectl -n meilisearch get pods
kubectl -n meilisearch get svc | grep read-lb
```
1. Record the read load balancer address. In the cluster it is normally:
```plaintext
http://meilisearch-read-lb.meilisearch.svc.cluster.local:7700
```
1. Check its health. This call needs no key:
```bash
kubectl -n run meili-check --rm -it --image=curlimages/curl --restart=Never -- \
curl -s http://meilisearch-read-lb.meilisearch.svc.cluster.local:7700/health
# {"status":"available"}
```
Use the Meilisearch asset’s deployment guide for replicas, storage and Redis
configuration.
### 2.2 Create two API keys
Create separate write and search-only keys. Assign only the search-only key to
case service.
| Key | Goes to | Actions | Indexes |
| --- | --- | --- | --- |
| Write key | `alpha-global-search-index-worker` | `documents.add`, `documents.delete`, `indexes.create`, `indexes.get`, `indexes.delete`, `settings.update` | `-cm-runtime-*` |
| Search-only key | `alpha-case-service` | `search` | `-cm-runtime-*` |
Create both with the master key through the standard Meilisearch keys API, or
follow the platform team’s procedure for the HA cluster:
```bash
MEILI=http://meilisearch-read-lb.meilisearch.svc.cluster.local:7700
PREFIX='' # see §3.2
curl -s -X POST "$MEILI/keys" -H "Authorization: Bearer " -H 'Content-Type: application/json' -d "{
\"description\": \"alpha $PREFIX index worker\",
\"actions\": [\"documents.add\",\"documents.delete\",\"indexes.create\",\"indexes.get\",\"indexes.delete\",\"settings.update\"],
\"indexes\": [\"$PREFIX-cm-runtime-*\"], \"expiresAt\": null }" | jq -r .key
curl -s -X POST "$MEILI/keys" -H "Authorization: Bearer " -H 'Content-Type: application/json' -d "{
\"description\": \"alpha $PREFIX case-service search\",
\"actions\": [\"search\"],
\"indexes\": [\"$PREFIX-cm-runtime-*\"], \"expiresAt\": null }" | jq -r .key
```
### 2.3 Store the keys as Kubernetes Secrets
Create both Secrets in the Alpha namespace:
```bash
kubectl -n create secret generic alpha-search-write-key --from-literal=api-key=''
kubectl -n create secret generic alpha-search-read-key --from-literal=api-key=''
```
* * *
## 3. Step B. Prepare the Alpha values
Configure the environment’s Trinity asset values or Helm values file. Supply
values explicitly when running Helm; the chart has no `values.yaml`.
### 3.1 Feature flags
To enable the preview, add `GLOBAL_SEARCH` to the environment’s complete `features.featureFlags` list.
An explicit list replaces the defaults. Preserve existing flags. For example:
```yaml
features:
featureFlags:
- ADMIN_FILTERS
- PULL_TASK
- INBOX_PROJECTION
- GLOBAL_CUSTOM_CODE
- BPM_CONFIG_VERSION
- CHANGE_TRACKING
- FALLBACK_TO_MAIN_CASEDEF
- CASE_FETCH_ALL_PERF_OPTIMIZATION # required by GLOBAL_SEARCH
- CASE_DATA_VERSIONING
- SNAPSHOT_CONTENT_JSONB
- GLOBAL_SEARCH # enable in Step D
```
The chart renders `FEATURE_FLAGS` as a JSON array. Services read it at startup.
A per-service `envs.FEATURE_FLAGS` override must be a literal JSON array string.
It can disable `GLOBAL_SEARCH` for that service but cannot enable it when the
deployment-wide list omits it. Keep `CASE_FETCH_ALL_PERF_OPTIMIZATION` as required
by section 1.4.
### 3.2 Search engine settings
```yaml
features:
search:
enabled: true
url: http://meilisearch-read-lb.meilisearch.svc.cluster.local:7700
indexPrefix: - # for example acme-prod
apiKey:
valueFrom:
secretKeyRef:
name: alpha-search-write-key
key: api-key
readApiKey:
valueFrom:
secretKeyRef:
name: alpha-search-read-key
key: api-key
```
`url` and `indexPrefix` become `MEILISEARCH_HOST` and `MEILISEARCH_INDEX_PREFIX`
on case service and the worker. Each receives its own `MEILISEARCH_API_KEY`:
`apiKey` for worker writes and `readApiKey` for case-service searches. Keys accept
a plain string or `valueFrom`.
Use a unique, stable prefix per environment. It may contain letters, digits,
hyphens and underscores, up to 100 characters. Index names are
`-cm-runtime-`.
With search enabled, the chart requires the URL, prefix and both keys.
`enabled: false` removes the worker and case-service Meilisearch settings but
keeps the API and producers. Use the worker pause setting in section 9.1 to pause
indexing. Remove obsolete `features.search.masterKey` and `meilisearch:` settings.
Do not change the URL or prefix while the queue has pending work. After changing
either, repeat Step F for every project.
### 3.3 Enquiry database and worker database role
Set the enquiry database and schema on case service. The chart uses these values
for the search service and worker:
```yaml
services:
backends:
alpha-case-service:
envs:
ENQUIRY_DB_NAME:
ENQUIRY_DB_SCHEMA_NAME:
```
To use a restricted worker role instead of the shared credentials in
`features.database`, set these overrides:
```yaml
services:
backends:
alpha-global-search-index-worker:
envs:
DB_USERNAME:
DB_PASSWORD:
valueFrom:
secretKeyRef: { name: alpha-search-worker-db, key: password }
```
Apply these grants after the queue tables exist. If Step D creates the tables,
apply the grants before resuming indexing in Step E. The search service’s database
role needs the same access for the backfill API. Grant access to new projection
tables when projects are added.
```sql
GRANT USAGE ON SCHEMA TO ;
GRANT SELECT ON .enquiry_config, .projection_data TO ;
GRANT SELECT ON ALL TABLES IN SCHEMA TO ; -- ci_* / ti_* projections
GRANT SELECT, INSERT, UPDATE, DELETE ON .search_index_jobs, .search_index_slots,
.search_index_blocks, .search_index_backfills TO ;
GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA TO ;
```
### 3.4 Service-to-service signing, session and ingress
- Configure `global.serviceAuth.internal.privateKey`, `publicKey` and `keyId`.
The worker requires the shared signing credentials and rejects service overrides.
- The search service uses the session, IDS and database settings in
`features.session`, `features.ids` and `features.database`.
- Keep `ingress.streaming: true` and `readTimeoutSeconds: 120`. Disable buffering
and compression for `/global-search/*` at every gateway. Allow at least 120
seconds of read and idle time.
- The chart sets `ALPHA_UI_GLOBAL_SEARCH_SERVICE_URL` to
`https:///global-search`. Override it only for a different route.
### 3.5 Optional search service overrides
Set overrides under `services.backends.`:
| Service | Setting | Default |
| --- | --- | --- |
| `alpha-global-search-service` | `version` | `global.imageTag` |
| `alpha-global-search-service` | `envs.GSS_SEARCH_TIMEOUT_MS` | 5000 |
| `alpha-global-search-service` | `envs.GSS_SUGGEST_TIMEOUT_MS` | 1500 |
| `alpha-global-search-service` | `envs.GSS_SSE_HEARTBEAT_MS` | 15000 |
| `alpha-global-search-index-worker` | `version` | API version, then `global.imageTag` |
Remove old `providers:` and `GLOBAL_SEARCH_PROVIDERS` overrides. The service
registers Inbox when `GLOBAL_SEARCH` is enabled.
* * *
## 4. Step C. Choose how to apply the queue migrations
Case service applies missing queue migrations through version 4 at startup when
`GLOBAL_SEARCH` is enabled, before listening for requests. This covers new and
existing enquiry schemas. These migrations are separate from
`migration-service-v2` and do not modify Meilisearch.
If the case-service database role can create and alter queue tables and indexes,
continue to Step D and verify section 4.3 afterward. Otherwise, use the Job in
section 4.2 with a suitable role before rollout.
The Job also avoids a startup race with `gts`, which checks for migration 2 but
does not apply it. If `gts` starts first, let it restart after migration completes.
| Version | Creates | Needed by |
| --- | --- | --- |
| 1 | `search_index_jobs`, `search_index_slots`, `search_index_blocks`, `search_index_migrations` | worker |
| 2 | Indexes for claim order, retention and metrics | case service and `gts` |
| 3 | `search_index_backfills` for backfill progress | backfill API |
| 4 | Descending-scan columns on `search_index_backfills` | backfill API in this release |
### 4.1 Baseline
```sql
SELECT version, applied_at
FROM .search_index_migrations ORDER BY version;
-- "relation does not exist" means no migration has run yet, expected on first install
```
### 4.2 Optional migration Job
Use the search service image tag for this release and the target enquiry database.
The database role must be able to create and alter the queue tables and indexes.
Save the following manifest as `gss-migrate-index-queue.yaml`:
```yaml
apiVersion: batch/v1
kind: Job
metadata:
name: gss-migrate-index-queue
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
imagePullSecrets: [{ name: }]
containers:
- name: migrate
image: //alpha-global-search-service:
command: ['node', 'main.js', '--migrate-index-queue']
env:
- { name: DB_HOST, value: '' }
- { name: DB_PORT, value: '5432' }
- { name: DB_USERNAME, value: '' }
- name: DB_PASSWORD
valueFrom: { secretKeyRef: { name: , key: password } }
- { name: ENQUIRY_DB_NAME, value: '' }
- { name: ENQUIRY_DB_SCHEMA_NAME, value: '' }
- { name: DB_SSL, value: 'false' } # plus DB_CA_CERT / DB_CA_CERT_PATH if SSL
```
```bash
kubectl -n apply -f gss-migrate-index-queue.yaml
kubectl -n wait --for=condition=complete job/gss-migrate-index-queue --timeout=10m
kubectl -n logs job/gss-migrate-index-queue
kubectl -n delete job gss-migrate-index-queue
```
The migration takes a PostgreSQL advisory lock. Concurrent migration runs wait
for that lock. Migration 2 locks tables while it builds indexes. Schedule a
maintenance window if an existing large queue still needs that migration.
### 4.3 Verify
Run these queries after case-service startup or the migration Job completes:
```sql
SELECT version FROM .search_index_migrations ORDER BY version;
-- expect 1, 2, 3, 4
SELECT to_regclass('.search_index_backfills') IS NOT NULL AS backfill_table; -- t
```
* * *
## 5. Step D. Deploy with indexing paused
Enable `GLOBAL_SEARCH` in the complete flag list from section 3.1. Pause the
worker with the override below. Producers continue adding changes to the queue
while the worker is paused.
```yaml
services:
backends:
alpha-global-search-index-worker:
envs:
GSS_INDEX_WORKER_ENABLED: 'false'
```
Apply through Trinity, or run `helm upgrade` with the customer values file last,
and wait for the rollout:
```bash
kubectl -n rollout status deploy/alpha-global-search-service
kubectl -n rollout status deploy/alpha-global-search-index-worker
kubectl -n rollout status deploy/alpha-case-service
kubectl -n rollout status deploy/gts
kubectl -n rollout status deploy/alpha-case-manager-ui
```
Checks:
1. Check the rendered flags in the case-service, `gts` and Case Manager pods.
For example:
```bash
kubectl -n exec deploy/alpha-case-service -- printenv FEATURE_FLAGS
# must contain both "CASE_FETCH_ALL_PERF_OPTIMIZATION" and "GLOBAL_SEARCH"
```
1. `GET /global-search/alive` is public and returns
`{"status":"ok","service":"alpha-global-search-service"}`.
2. The search service startup log lists the providers:
`1 of 1 provider(s) registered: inbox`.
3. `GET /caseservice/ping-meili` returns `meilisearch: "ok"`.
`meilisearch: "unavailable"` indicates a URL, key or network problem.
`meilisearch: "disabled"` means case service has not enabled the search client.
4. The worker is up but paused. `kubectl logs deploy/alpha-global-search-index-worker`
shows no claims, and `/global-search/ready` returns 200 inside the cluster.
5. Verify queue migrations 1 through 4 using section 4.3. Apply any database grants
from section 3.3 before resuming indexing.
* * *
## 6. Step E. Resume indexing
1. Set `GSS_INDEX_WORKER_ENABLED` to `"true"`, or remove the override to use that
default:
```yaml
services:
backends:
alpha-global-search-index-worker:
envs:
GSS_INDEX_WORKER_ENABLED: 'true'
```
1. Apply, then wait for `rollout status deploy/alpha-global-search-index-worker`.
2. Hard-refresh Case Manager. The search box appears in the header. It finds only
what the worker has indexed so far. Step F indexes the history.
Every service reads the flag once at startup. A change to `features.featureFlags`
takes effect only after Helm restarts the pods. Never edit `FEATURE_FLAGS` inside a
running pod.
* * *
## 7. Step F. Backfill each project
The backfill indexes one project’s existing enquiry rows, the `ci_*` cases and
their `ti_*` tasks, through the queue. It never reads Case Manager or BPM. It
queues one case, waits until Meilisearch confirms the case is visible, then moves
the cursor. Requests process cases sequentially, regardless of `batchSize`.
Use the base URL `/global-search/migration/search-index-backfill` for the
`POST` endpoints below. Include the session cookie and `x-csrf-token` from section
1.2.
### 7.1 When to run it
| Situation | Action |
| --- | --- |
| First deployment with existing cases | Run `up` for every project. |
| Search was disabled, the engine was replaced or emptied, or the prefix changed | Run `up` with `resetCursor: true` for every project. |
| A project’s index needs rebuilding | Run `down`, then `up`. See section 9.3. |
| A new project or enquiry projection repair | No separate search backfill. Writes and repairs queue their changes while search is enabled. |
| Searchable fields changed | The worker updates settings. Run `up` with `resetCursor: true` to rebuild existing documents immediately. |
### 7.2 Read the baseline with `POST /status/project`
```json
{ "projectId": "" }
```
Check `caseType`, `totalCases`, `remainingCases` and `job`. A project with no
saved backfill has `job: null`. If the API reports missing enquiry configuration,
complete the projection backfill in section 1.4.
### 7.3 Preview with `POST /up/project`
```json
{ "projectId": "", "dryRun": true }
```
The response has the same counts plus `hasMore`, `order` and `dryRun: true`. A
dry run queues nothing and leaves progress unchanged.
### 7.4 Run the backfill with `POST /up/project`
Start a new scan with one case:
```json
{ "projectId": "", "resetCursor": true, "batchSize": 1, "maxBatches": 1 }
```
A completed one-case request returns `processedThisRun: 1`. It can return
`202 pending` if indexing is still in progress. Continue with the request below,
without `resetCursor`, until `hasMore` is `false`:
```json
{ "projectId": "", "batchSize": 100, "maxBatches": 20, "batchDelayMs": 200 }
```
Check `hasMore` after each call. `processedThisRun` counts confirmed cases and
`durationMs` reports elapsed time.
A request waits up to 30 seconds for indexing across its batches. HTTP 202 with
`status: "pending"` and `hasMore: true` means indexing is incomplete. Retry without
`resetCursor`, including after a timeout or disconnect. The service saves progress
and reuses an unexpired pending ticket. It can enqueue the case again after the
ticket expires.
Request fields:
| Field | Default | Range | Meaning |
| --- | --- | --- | --- |
| `projectId` | none | required | Project to index |
| `order` | `desc` | `asc` or `desc` | Case ID order. `desc` indexes the newest cases first |
| `batchSize` | 100 | 1 to 1000 | Cases read per batch |
| `maxBatches` | 20 | 1 to 100 | Batches per request. The 30-second limit usually ends the request first |
| `batchDelayMs` | 200 | 0 to 5000 | Pause between batches, in milliseconds |
| `resetCursor` | false | true or false | Start a new scan from the top. Use it once at the start, never on a retry |
| `dryRun` | false | true or false | Preview only |
A retry without `order` keeps the saved order. To change the order, send
`resetCursor: true`, or the service returns `409`. A descending scan records the
highest case ID when it starts. Normal writes index cases created after that, so
the scan never sees them. To scan again after a scan completes, send
`resetCursor: true`.
### 7.5 Verify with `POST /status/project`
Completion requires `remainingCases: 0` and `job.status: "completed"`.
`job.status` is one of `idle`, `running`, `pending`, `completed` or `failed`.
Retry a pending request. For a failed request, inspect `lastError`. Verify the
saved progress in SQL:
```sql
SELECT project_id, operation, status, cases_processed, sort_order, cursor_cid, snapshot_max_cid, last_error, updated_at
FROM .search_index_backfills WHERE project_id = '';
```
Then search Case Manager for the ID of an old case and for a field value from an
old case.
### 7.6 Several projects
Run only one backfill request per project at a time. Concurrent requests can
return `409 Backfill progress changed in another request`. Different projects
share the queue’s indexing slots, which default to four. Check worker capacity
and live indexing load before increasing concurrency.
* * *
## 8. Step G. Verify the deployment
1. Run the smoke script from an alpha-core checkout:
```bash
node scripts/smoke/global-search-smoke.mjs --url https:///global-search
node scripts/smoke/global-search-smoke.mjs --url https:///global-search --cookie 'alpha.sessionid=' --project
```
The script checks `/alive`, the 401 answers without a session, that `/configs`
lists `inbox`, and that the stream is uncompressed `text/event-stream` in several
chunks. If results arrive only at the end, check gateway buffering and compression
as described in section 3.4.
1. Check that no queue job is dead or stuck:
```sql
SELECT status, COUNT(*) FROM .search_index_jobs GROUP BY status;
-- pending and processing should fall to about 0; dead should be 0
SELECT id, project_id, cid, kind, attempts, last_error, updated_at
FROM .search_index_jobs WHERE status = 'dead' ORDER BY updated_at DESC LIMIT 20;
SELECT * FROM .search_index_blocks; -- projects or branches blocked by missing config
```
1. Scrape the worker’s internal `/global-search/metrics`. It reports queue depth
and age, active slots, the largest indexing lag over the last 5 minutes,
failures, and writes the cluster accepted but has not yet confirmed.
2. Claim or reassign a task in Case Manager, then confirm search finds the change
within a few seconds.
3. In the UI, the search box shows “Case Manager” results with My Tasks, Group
Tasks, Team Tasks and Enquiry sections, filtered to the user’s permissions.
A healthy `/alive` or a successful Helm rollout does not prove that the history is
indexed.
* * *
## 9. Operations, rollback and troubleshooting
### 9.1 Pause and resume indexing
To pause without losing work, keep `GLOBAL_SEARCH` on and set
`alpha-global-search-index-worker.envs.GSS_INDEX_WORKER_ENABLED: "false"`. Producers
keep adding to the queue, the worker claims nothing new, active jobs finish and
cleanup still runs. Backfill requests return `202 pending` while indexing is
paused. Set the value to `"true"` to resume.
### 9.2 Turn search off and on again
To turn search off, remove `GLOBAL_SEARCH` from `features.featureFlags` and deploy.
The chart removes the API and the worker, Case Manager hides the box, and the
producers stop adding to the queue, so nothing records the changes made while
search is off. Keep the queue tables and the Meilisearch indexes.
To re-enable search, restore the flag and follow Steps D and E. Then run Step F with
`resetCursor: true` for every project to index the changes made while it was off.
### 9.3 Rebuild a project index with `POST /down/project`
```json
{ "projectId": "" }
```
This call deletes only the project’s Meilisearch index and its search backfill
progress. It leaves `ci_*`, `ti_*`, `si_*` and the source data alone. On
`202 pending`, retry until the answer is
`"Search index deleted; enquiry projections are unchanged"`. Then run `up` as described in section 7.4.
While a deletion is pending, `up` returns `409`. A write after the delete can
create the index again right away.
> Do not use the case-service `.../enquiry-projection-backfill/down/project` for
> search. It truncates the enquiry tables and leaves the search index alone.
### 9.4 Dead jobs and capacity
Run these from an alpha-core checkout with the enquiry DB environment variables
set:
```bash
node --import tsx scripts/index-queue.ts replay # reopen a dead job, then reconcile
node --import tsx scripts/index-queue.ts capacity # global indexing slots, 1 to 64; default 4; drain before shrinking
node --import tsx scripts/index-queue.ts migrate # same as the migration Job
```
The worker marks a job dead after its fifth failed attempt. The wait between
attempts starts at 30 seconds and grows to at most 5 minutes. Fix the cause, such
as the key, the network or the configuration, before replaying the job.
### 9.5 Roll back the application
Before reverting images, check the target release’s rollback requirements. Keep
the queue schema and Meilisearch indexes. Do not drop the `search_index_*` tables
or resume a descending
backfill with an image older than this release, because older images only support
ascending scans.
### 9.6 Troubleshooting
| Symptom | Check or action |
| --- | --- |
| No API or worker Deployment | Check global and service flags. For the worker, check `features.search.enabled`. |
| Helm reports missing Meilisearch or enquiry settings | Complete sections 3.2 and 3.3. Keys must be strings or `valueFrom.secretKeyRef` maps. |
| Case service or `gts` reports missing migration 2 | Check case-service migration logs and database permissions. Verify section 4.3, then restart. Use the optional Job if needed. |
| Search box disabled | Complete section 1.4 and restore `CASE_FETCH_ALL_PERF_OPTIMIZATION`. |
| Search box missing | Check Case Manager’s image and `FEATURE_FLAGS`, then hard-refresh. |
| Results arrive together or search times out | Disable gateway buffering and compression. Use a read timeout of at least 120 seconds. |
| Meilisearch unavailable or `INDEX_UNAVAILABLE` | Check the URL, search-only key and network policies. |
| Worker reports `UNAUTHORIZED` | Check write-key actions and the `-cm-runtime-*` index pattern. |
| Empty results without an error | Check whether the project has an index. A missing index returns no results. Run Step F. |
| Backfill returns 400 for missing enquiry configuration | Save the inbox configuration or complete the projection backfill. |
| Backfill returns 409 | Finish concurrent requests or pending deletion. Changing scan order requires `resetCursor`. |
| Backfill stays pending | Check worker pause settings, logs and dead jobs. |
| Backfill returns 503 | Check `GLOBAL_SEARCH` and `ENQUIRY_DB_NAME`. |
| Backfill returns 401 | Log in again and obtain a new session cookie and CSRF token. |
| `search_index_blocks` contains rows | Restore the missing project or branch inbox configuration. The worker checks again each minute. |
* * *
## 10. API reference
Search service base URL: `/global-search`. Use a session cookie and CSRF
token for authenticated `POST` requests.
| Method | Path | Body |
| --- | --- | --- |
| GET | `/alive` | None. Public. |
| GET | `/configs` | None. |
| POST | `/search`, `/suggest` | `{ q, projectId, providers? }`. Returns SSE. |
| POST | `/migration/search-index-backfill/status/project` | `{ projectId }` |
| POST | `/migration/search-index-backfill/up/project` | Fields in section 7.4. |
| POST | `/migration/search-index-backfill/down/project` | `{ projectId }` |
Worker probes and metrics use `GET /alive`, `/ready` and `/metrics` at
`:8080/global-search` inside the cluster. Case-service health uses public
`GET /caseservice/ping-meili`.