Monitorar o Openflow¶

Acesso aos logs do Openflow¶

O Snowflake envia os logs do Openflow para a tabela de eventos que você definiu ao configurar o Openflow. A Snowflake recomenda que você inclua um carimbo de data/hora na cláusula WHERE das consultas da tabela de eventos. Isso é especialmente importante devido ao volume potencial de dados gerados por vários componentes do Snowflake. Ao aplicar filtros, você pode recuperar um subconjunto menor de dados, o que melhora o desempenho da consulta.

A tabela de eventos inclui as seguintes colunas, que fornecem informações úteis sobre os logs coletados pelo Snowflake a partir do Openflow:

• TIMESTAMP: mostra quando o Snowflake coletou o registro.

• RESOURCE_ATTRIBUTES: fornece um objeto JSON que identifica o serviço Snowflake que gerou a mensagem de registro. Por exemplo, ele fornece detalhes como o aplicativo e o ID de plano de dados para o Openflow.

  {
  "application": "openflow",
  "cloud.service.provider": "aws",
  "k8s.container.name": "pg-dev-server",
  "k8s.container.restart_count": "0",
  "k8s.namespace.name": "runtime-pg-dev",
  "k8s.node.name": "ip-10-10-62-36.us-east-2.compute.internal",
  "k8s.pod.name": "pg-dev-0",
  "k8s.pod.start_time": "2025-04-25T22:14:29Z",
  "k8s.pod.uid": "94610175-1685-4c8f-b0a1-42898d1058e6",
  "k8s.statefulset.name": "pg-dev",
  "openflow.dataplane.id": "abeddb4f-95ae-45aa-95b1-b4752f30c64a"
  }
  

  Copy

• RECORD_ATTRIBUTES: para um serviço Snowflake, identifica uma fonte de erro (saída padrão ou erro padrão).

  {
  "log.file.path": "/var/log/pods/runtime-pg-dev_pg-dev-0_94610175-1685-4c8f-b0a1-42898d1058e6/pg-dev-server/0.log",
  "log.iostream": "stdout",
  "logtag": "F"
  }
  

  Copy

• VALUE: a saída padrão e o erro padrão são divididos em linhas, com cada linha gerando seu próprio registro na tabela de eventos.

  "{\"timestamp\":1746655642080,\"nanoseconds\":80591397,\"level\":\"INFO\",\"threadName\":\"Clustering Tasks Thread-2\",\"loggerName\":\"org.apache.nifi.controller.cluster.ClusterProtocolHeartbeater\",\"formattedMessage\":\"Heartbeat created at 2025-05-07T22:07:22.071Z and sent to pg-dev-0.pg-dev.runtime-pg-dev.svc.cluster.local:8445 at 2025-05-07T22:07:22.080590784Z; determining Cluster Coordinator took 7 millis; DNS lookup for coordinator took 0 millis; connecting to coordinator took 0 millis; sending heartbeat took 1 millis; receiving first byte from response took 1 millis; receiving full response took 1 millis; total time was 9 millis\",\"throwable\":null}"
  

SELECT
    timestamp,
    resource_attributes:"k8s.namespace.name" AS runtime_key,
    parse_json(value::string):loggerName AS logger,
    parse_json(value::string):formattedMessage AS log_value
FROM openflow.telemetry.EVENTS_<account-id>
WHERE true
AND timestamp < dateadd('days', -1, sysdate())
AND record_type = 'LOG'
AND resource_attributes:"k8s.namespace.name" LIKE 'runtime-%'
AND resource_attributes:"k8s.container.name" LIKE '%-server'
AND parse_json(value::string):level = 'ERROR'
ORDER BY timestamp desc
LIMIT 5;

SELECT
    timestamp,
    RESOURCE_ATTRIBUTES:"k8s.namespace.name" AS Namespace,
    RESOURCE_ATTRIBUTES:"k8s.pod.name" AS Pod,
    RESOURCE_ATTRIBUTES:"k8s.container.name" AS Container,
    value
FROM openflow.telemetry.EVENTS_<account-id>
WHERE true
AND record_type = 'LOG'
AND timestamp > dateadd(minute, -5, sysdate())
AND value LIKE '%Caused By%'
ORDER BY timestamp desc
LIMIT 10;

SELECT
    timestamp,
    RECORD_ATTRIBUTES:component AS Processor,
    RECORD_ATTRIBUTES:id AS Processor_ID,
    value AS Running
FROM openflow.telemetry.EVENTS_<account-id>
WHERE true
AND RECORD:metric:name = 'processor.run.status.running'
AND RECORD_TYPE='METRIC'
AND timestamp > dateadd(minute, -5, sysdate());

SELECT
    timestamp,
    RESOURCE_ATTRIBUTES:"k8s.namespace.name" AS Namespace,
    RESOURCE_ATTRIBUTES:"k8s.pod.name" AS Pod,
    RESOURCE_ATTRIBUTES:"k8s.container.name" AS Container,
    value AS CPU_Usage
FROM openflow.telemetry.EVENTS_<account-id>
WHERE TIMESTAMP > dateadd(minute, -1, sysdate())
AND RECORD_TYPE = 'METRIC'
AND RECORD:metric:name ilike 'container.cpu.usage'
AND RESOURCE_ATTRIBUTES:"k8s.namespace.name" ilike 'runtime%'
AND RESOURCE_ATTRIBUTES:"k8s.container.name" ilike '%server'
AND RESOURCE_ATTRIBUTES:"k8s.namespace.name" NOT IN ('runtime-infra', 'runtime-operator')
ORDER BY TIMESTAMP desc, CPU_Usage desc;

Métricas disponíveis¶

SELECT * from events
WHERE
1 = 1
AND record_type = 'METRIC'
AND resource_attributes:application = 'openflow'
AND record:metric.name IN ('cores.load', 'cores.available')
ORDER BY timestamp desc
LIMIT 1000;