Re-mining templates

Templates reflect the settings they were mined with. After you change a Drain or masking setting (logminer.sim_threshold, logminer.max_clusters_per_service or logminer.keep_http_status), only logs mined from then on follow the new settings. tayga-devtools remine rebuilds the templates from the stored logs of the last 3 days, with the logminer’s own code and its current settings, without raising alerts for the rebuilt templates.

What it does

Truncates log_templates, log_template_hits and log_template_minutes.
Reads logs from the last 3 days in (ts, log_id) order, 10,000 rows per page, and mines them with the current [logminer] settings, read from the TAYGA_CONFIG file and TAYGA__LOGMINER__* variables like the logminer itself.
After each page, writes the hits and the changed templates; log_template_minutes is filled through its materialized view.
Stores in logminer_state the watermark (the newest mined log), a new masking epoch starting now, and the current masking version.

The watermark keeps the restarted logminer from announcing the rebuilt templates as new. The new epoch adds the 15-minute warmup: no new alert fires for templates first seen in the 15 minutes after the re-mine. log_alerts and the silence settings are kept.

Running it

remine is part of tayga-devtools, which ships in the Tayga image and can also be built from source. It talks to ClickHouse directly, with the [clickhouse] settings of its config (by default http://localhost:18123, database tayga); --clickhouse and --database override them. Inside the logminer or API container it picks up the same settings as those services.

In every setup the steps are the same: preview with a dry run, stop every logminer replica, wait about 3 minutes (a real run refuses while any logminer heartbeat is under 3 minutes old), re-mine, and start the logminer again.

In the directory of the standalone bundle:

docker compose exec tayga-logminer tayga-devtools remine --dry-run   # read-only, allowed while the logminer runs
docker compose stop tayga-logminer
# wait about 3 minutes
docker compose run --rm --no-deps tayga-logminer tayga-devtools remine
docker compose start tayga-logminer

docker compose run starts a one-off container from the logminer’s service definition, so the re-mine uses the logminer’s own settings, including any TAYGA__LOGMINER__* variables.

With the release name tayga from the install guide (the deployments are <release>-…, or <release>-tayga-… when the release name does not contain tayga):

kubectl --namespace tayga exec deploy/tayga-logminer -- tayga-devtools remine --dry-run
kubectl --namespace tayga scale deploy/tayga-logminer --replicas 0
# wait about 3 minutes
kubectl --namespace tayga exec deploy/tayga-api -- tayga-devtools remine
kubectl --namespace tayga scale deploy/tayga-logminer --replicas 1   # or your logminer.replicas

The real run goes through the API pod, which has the same ClickHouse settings, because the logminer pods are scaled to zero. If you set TAYGA__LOGMINER__* variables in logminer.extraEnv, pass the same ones: kubectl … exec deploy/tayga-api -- env TAYGA__LOGMINER__SIM_THRESHOLD=0.5 tayga-devtools remine.

Next to the OpenTelemetry demo, from the repository root (this is the Makefile’s compose command):

cargo run --release -q -p tayga-devtools -- remine --dry-run

TAYGA_ROOT=$PWD docker compose --project-directory vendor/opentelemetry-demo \
  -f vendor/opentelemetry-demo/compose.yaml -f vendor/opentelemetry-demo/compose.full.yaml \
  -f vendor/opentelemetry-demo/compose.observability.yaml \
  -f deploy/compose.infra.yaml -f deploy/compose.tayga.yaml stop tayga-logminer
# wait about 3 minutes
cargo run --release -q -p tayga-devtools -- remine

Then start the logminer with the same compose command and start tayga-logminer. Build with --release: on 8.6 million logs a debug-build dry run took 501 s and a release-build real run 211 s (2026-10-06). If the logminer runs with non-default TAYGA__LOGMINER__* variables, pass the same ones to cargo run.

The dry run prints templates per service before and after, the template ids added, removed and unchanged, and the silence settings that would be orphaned. Nothing checks that the re-mine and the logminer use the same settings.

Things to know

Downtime. The logminer is down for the run plus the 3-minute wait: 211 s for 8.6 million logs in the run below.
The heartbeat guard. Each replica writes logminer_heartbeat_ns:<replica id> on every detection pass. A real run reads all of them and refuses while the newest is under 3 minutes old: Error: the logminer looks alive (heartbeat 43s ago, limit 180s). Stop it first with .... --force skips the check; use it only when you know the logminer is down. --dry-run never checks.
Not atomic. The tables are truncated before mining. If a run fails partway, run it again before you start the logminer.
Only 3 days. Only stored logs are mined. Templates whose logs are all older are dropped, although log_templates itself keeps rows for 30 days.
Template ids can change. An id hashes the service and the template as first seen, so it is stable only for the same logs, order and settings. After a settings change many ids can change:
- alerts keep their own template text, but their links can point at ids that no longer exist;
- silence settings on vanished ids are reported as orphaned and left in place; switch silence on again for the new ids.
Spikes are not gated. A template that is spiking when its id changes gets a new spike alert id, and the notifier may deliver it a second time.
An empty window. With no logs in the last 3 days the template tables end up empty and the watermark is unchanged.

A real run

On the demo stack on 2026-10-06, the release-build run read and mined 8,647,835 logs in 211.1 s: templates went from 415 to 402 (27 added, 40 removed, 375 unchanged), no silence setting was orphaned, and no alert of any kind fired in the 15 minutes after the logminer restarted.