How to filter and reduce the volume of Legacy metrics published by io.Connect Desktop

io.Insights
, ,
1

Applies to: legacy Metrics API

Note: the legacy Metrics API is deprecated

The legacy Metrics API described above is deprecated in favor of the OpenTelemetry-based io.Insights - see the Legacy io.Insights overview, which labels itself as covering the deprecated implementation. (The documentation states the deprecation but does not tie it to a specific version.)
We strongly recommend moving to io.Insights.

Background

io.Connect Desktop can publish a set of default metrics through the io.Connect Gateway, including periodic publisher status (“heartbeat”) messages. In some deployments the legacy metrics pipeline produces more data than the receiving infrastructure needs. If you are using the default (legacy) metrics and are not yet on OpenTelemetry, you can control what gets published through the Gateway’s metrics publishing and filtering configuration.

The Gateway metrics publishing and filtering configuration is located under gw.configuration.metrics in the system.json file.

Reference: Legacy io.Insights — Overview (Publishing & Filtering Metrics).

Q: How can I reduce which metrics are published?

Use the filters configuration. Filtering rules first match the identity of a metric publisher; for every matching publisher rule, the metric is then checked against that rule’s whitelist and blacklist:

  • A metric is excluded if a blacklist entry matches it.
  • When a whitelist is present, the metric must match it to be published.
  • The non-matched setting ("whitelist" or "blacklist") determines what happens to publishers that match no publisher rule.
    For Interop metrics, additional filtering is available at the Interop method and argument level. For the exact schema and copy-paste examples, follow the reference page: Legacy io.Insights — Overview. Use the configuration exactly as shown there.

Q: What controls the frequency of publisher “heartbeat” messages?

The file, rest, and kafka publishers support a heartbeats property that sets the interval (in milliseconds) of the publisher’s status messages. This controls the metrics pipeline’s own status messages - see the reference page for the exact property placement and defaults. To exclude specific application metrics from being published, use the whitelist/blacklist filtering described above.

Note: io.Insights as replacement for Legacy metrics

The legacy Metrics API and Gateway metrics pipeline described above are deprecated and replaced by the OpenTelemetry-based io.Insights. When moving note the following points:

  • io.Insights supports OpenTelemetry metrics, traces, and logs, and supports custom metrics.
  • The legacy gw.configuration.metrics filters do not directly carry over to io.Insights; io.Insights has its own filtering configuration.
  • Breaking change in 10.1: the defaultMetricsEnabled property under otel.metrics was renamed to platformMetricsEnabled.

For the modern approach, see the io.Insights documentation and the io.Insights configuration for io.Connect Desktop. The io.Connect Desktop 10.1 changelog lists the related changes.