Apache SkyWalking – Nginx

Zh: SkyWalking 如何支持 ZIO 等 Scala Effect Runtime

Thu, 04 Jan 2024 00:00:00 +0000

背景介绍

在 Scala 中，纯函数式中主要使用 Fiber，而不是线程，诸如 Cats-Effect、ZIO 等 Effect 框架。您可以将 Fiber 视为轻量级线程，它是一种并发模型，由框架本身掌控控制权，从而消除了上下文切换的开销。基于这些 Effect 框架开发的 HTTP、gRCP、GraphQL 库而开发的应用，我们一般称为 纯函数式应用程序。

我们以 ZIO 为切入点，演示 SkyWalking Scala 如何支持 Effect 生态。

ZIO Trace

首先，我们想要实现 Fiber 上下文传递，而不是监控 Fiber 本身。对于一个大型应用来说，可能存在成千上万个 Fiber，监控 Fiber 本身的意义不大。

虽然 Fiber 的 Span 是在活跃时才会创建，但难免会有目前遗漏的场景，所以提供了一个配置 plugin.ziov2.ignore_fiber_regexes。它将使用正则去匹配 Fiber location，匹配上的 Fiber 将不会创建 Span。

Fiber Span的信息如下：

下面是我们使用本 ZIO 插件，和一些官方插件（hikaricp、jdbc、pulsar）完成的 Trace：

分析

在 ZIO 中，Fiber可以有两种方式被调度，它们都是 zio.Executor 的子类。当然您也可以使用自己的线程池，这样也需被 ZIO 包装，其实就类似下面的 blockingExecutor。

abstract class Executor extends ExecutorPlatformSpecific {
  self =>
  def submit(runnable: Runnable)(implicit unsafe: Unsafe): Boolean
}

一种是系统默认线程池 defaultExecutor：

private[zio] trait RuntimePlatformSpecific {
  final val defaultExecutor: Executor =
    Executor.makeDefault()
}

另一种是专用于阻塞 IO 的线程池 blockingExecutor：

private[zio] trait RuntimePlatformSpecific {
  final val defaultBlockingExecutor: Executor =
    Blocking.blockingExecutor
}

默认线程池 `defaultExecutor`

对于 defaultExecutor，其本身是很复杂的，但它就是一个 ZIO 的 Fiber 调度（执行）器：

/**
 * A `ZScheduler` is an `Executor` that is optimized for running ZIO
 * applications. Inspired by "Making the Tokio Scheduler 10X Faster" by Carl
 * Lerche. [[https://tokio.rs/blog/2019-10-scheduler]]
 */
private final class ZScheduler extends Executor

由于它们都是 zio.Executor 的子类，我们只需要对其及其子类进行增强：

final val ENHANCE_CLASS = LogicalMatchOperation.or(
  HierarchyMatch.byHierarchyMatch("zio.Executor"),
  MultiClassNameMatch.byMultiClassMatch("zio.Executor")
)

它们都是线程池，我们只需要在 zio.Executor 的 submit 方法上进行类似 ThreadPoolExecutor 上下文捕获的操作，可以参考 jdk-threadpool-plugin

这里需要注意，因为 Fiber 也是一种 Runnable：

private[zio] trait FiberRunnable extends Runnable {
  def location: Trace
  def run(depth: Int): Unit
}

zio-v2x-plugin

阻塞线程池 `blockingExecutor`

对于 blockingExecutor，其实它只是对 Java 线程池进行了一个包装：

object Blocking {

  val blockingExecutor: zio.Executor =
    zio.Executor.fromThreadPoolExecutor {
      val corePoolSize  = 0
      val maxPoolSize   = Int.MaxValue
      val keepAliveTime = 60000L
      val timeUnit      = TimeUnit.MILLISECONDS
      val workQueue     = new SynchronousQueue[Runnable]()
      val threadFactory = new NamedThreadFactory("zio-default-blocking", true)

      val threadPool = new ThreadPoolExecutor(
        corePoolSize,
        maxPoolSize,
        keepAliveTime,
        timeUnit,
        workQueue,
        threadFactory
      )

      threadPool
    }
}

由于其本身是对 ThreadPoolExecutor 的封装，所以，当我们已经实现了 zio.Executor 的增强后，只需要使用官方 jdk-threadpool-plugin 插件即可。这里我们还想要对代码进行定制修改和复用，所以重新使用 Scala 实现了一个 executors-plugin 插件。

串连 Fiber 上下文

最后，上面谈到过，Fiber 也是一种 Runnable，因此还需要对 zio.internal.FiberRunnable 进行增强。大致分为两点，其实与 jdk-threading-plugin 是一样的。

每次创建 zio.internal.FiberRunnable 实例时，都需要保存现场，即构造函数增强。
每次运行时创建一个过渡的 Span，将当前线程上下文与之前保存在构造函数中的上下文进行关联。Fiber 可能被不同线程执行，所以这是必须的。

zio-v2x-plugin

说明

当我们完成了对 ZIO Fiber 的上下文传播处理后，任意基于 ZIO 的应用层框架都可以按照普通的 Java 插件思路去开发。我们只需要找到一个全局切入点，这个切入点应该是每个请求都会调用的方法，然后对这个方法进行增强。

要想激活插件，只需要在 Release Notes 下载插件，放到您的 skywalking-agent/plugins 目录，重新启动服务即可。

如果您的项目使用 sbt assembly 打包，您可以参考这个示例。该项目使用了下列技术栈：

    libraryDependencies ++= Seq(
      "io.d11"               %% "zhttp"                % zioHttp2Version,
      "dev.zio"              %% "zio"                  % zioVersion,
      "io.grpc"               % "grpc-netty"           % "1.50.1",
      "com.thesamet.scalapb" %% "scalapb-runtime-grpc" % scalapb.compiler.Version.scalapbVersion
    ) ++ Seq(
      "dev.profunktor" %% "redis4cats-effects"  % "1.3.0",
      "dev.profunktor" %% "redis4cats-log4cats" % "1.3.0",
      "dev.profunktor" %% "redis4cats-streams"  % "1.3.0",
      "org.typelevel"  %% "log4cats-slf4j"      % "2.5.0",
      "dev.zio"        %% "zio-interop-cats"    % "23.0.03",
      "ch.qos.logback"  % "logback-classic"     % "1.2.11",
      "dev.zio"        %% "zio-cache"           % zioCacheVersion
    )

Blog: Monitoring Nginx with SkyWalking

Sat, 23 Dec 2023 00:00:00 +0000

Background

Apache SkyWalking is an open-source application performance management system that helps users collect and aggregate logs, traces, metrics, and events, and display them on the UI.

In order to achieve monitoring capabilities for Nginx, we have introduced the Nginx monitoring dashboard in SkyWalking 9.7, and this article will demonstrate the use of this monitoring dashboard and introduce the meaning of related metrics.

Setup Monitoring Dashboard

Metric Define and Collection

Since nginx-lua-prometheus is used to define and expose metrics, we need to install lua_nginx_module for Nginx, or use OpenResty directly.

In the following example, we define four metrics via nginx-lua-prometheus and expose the metrics interface via nginx ip:9145/metrics:

histogram: nginx_http_latency，monitoring http latency
gauge: nginx_http_connections，monitoring nginx http connections
counter: nginx_http_size_bytes，monitoring http size of request and response
counter: nginx_http_requests_total，monitoring total http request numbers

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    lua_shared_dict prometheus_metrics 10M;
    # lua_package_path "/path/to/nginx-lua-prometheus/?.lua;;";

    init_worker_by_lua_block {
      prometheus = require("prometheus").init("prometheus_metrics")

      metric_bytes = prometheus:counter(
        "nginx_http_size_bytes", "Total size of HTTP", {"type", "route"})
      metric_requests = prometheus:counter(
        "nginx_http_requests_total", "Number of HTTP requests", {"status", "route"})
      metric_latency = prometheus:histogram(
        "nginx_http_latency", "HTTP request latency", {"route"})
      metric_connections = prometheus:gauge(
        "nginx_http_connections", "Number of HTTP connections", {"state"})
    }

    server {
        listen 8080;

        location /test {
          default_type application/json;
          return 200  '{"code": 200, "message": "success"}';

          log_by_lua_block {
            metric_bytes:inc(tonumber(ngx.var.request_length), {"request", "/test/**"})
            metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", "/test/**"})
            metric_requests:inc(1, {ngx.var.status, "/test/**"})
            metric_latency:observe(tonumber(ngx.var.request_time), {"/test/**"})
          }
        }
    }

    server {
      listen 9145;
      location /metrics {
        content_by_lua_block {
          metric_connections:set(ngx.var.connections_reading, {"reading"})
          metric_connections:set(ngx.var.connections_waiting, {"waiting"})
          metric_connections:set(ngx.var.connections_writing, {"writing"})
          prometheus:collect()
        }
      }
    }
}

In the above example, we exposed the route-level metrics, and you can also choose to expose the host-level metrics according to the monitoring granularity:

http {
  log_by_lua_block {
      metric_bytes:inc(tonumber(ngx.var.request_length), {"request", ngx.var.host})
      metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", ngx.var.host})
      metric_requests:inc(1, {ngx.var.status, ngx.var.host})
      metric_latency:observe(tonumber(ngx.var.request_time), {ngx.var.host})
  }
}

or upstream-level metrics：

upstream backend {
  server ip:port;
}

server {
  
  location /test_upstream {
  
    proxy_pass http://backend;
  
    log_by_lua_block {
      metric_bytes:inc(tonumber(ngx.var.request_length), {"request", "upstream/backend"})
      metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", "upstream/backend"})
      metric_requests:inc(1, {ngx.var.status, "upstream/backend"})
      metric_latency:observe(tonumber(ngx.var.request_time), {"upstream/backend"})
    }
  }
}

After defining the metrics, we start nginx and opentelemetry-collector to collect the metrics and send them to the SkyWalking backend for analysis and storage.

Please ensure that job_name: 'nginx-monitoring', otherwise the reported data will be ignored by SkyWalking. If you have multiple Nginx instances, you can distinguish them using the service and service_instance_id labels：

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'nginx-monitoring'
          scrape_interval: 5s
          metrics_path: "/metrics"
          static_configs:
            - targets: ['nginx:9145']
              labels:
                service: nginx
                service_instance_id: nginx-instance
processors:
  batch:

exporters:
  otlp:
    endpoint: oap:11800
    tls:
      insecure: true
service:
  pipelines:
    metrics:
      receivers:
        - prometheus
      processors:
        - batch
      exporters:
        - otlp

If everything goes well, you will see the metric data reported by Nginx under the gateway menu of the skywalking-ui:

Access & Error Log Collection

SkyWalking Nginx monitoring provides log collection and error log analysis. We can use fluent-bit to collect and report access logs and error logs to SkyWalking for analysis and storage.

Fluent-bit configuration below defines the log collection directory as /var/log/nginx/. The access and error logs will be reported through rest port 12800 of oap after being processed by rewrite_access_log and rewrite_error_log functions:

[SERVICE]
    Flush          5
    Daemon         Off
    Log_Level      warn
[INPUT]
    Name           tail
    Tag            access
    Path           /var/log/nginx/access.log
[INPUT]
    Name           tail
    Tag            error
    Path           /var/log/nginx/error.log
[FILTER]
    Name           lua
    Match          access
    Script         fluent-bit-script.lua
    Call           rewrite_access_log
[FILTER]
    Name           lua
    Match          error
    Script         fluent-bit-script.lua
    Call           rewrite_error_log
[OUTPUT]
    Name            stdout
    Match           *
    Format          json
[OUTPUT]
    Name            http
    Match           *
    Host            oap
    Port            12800
    URI             /v3/logs
    Format          json

In the fluent-bit-script.lua, we use LOG_KIND tag to distinguish between access logs and error logs.

To associate with the metrics, please ensure that the values of service and serviceInstance are consistent with the metric collection definition in the previous section.

function rewrite_access_log(tag, timestamp, record)
    local newRecord = {}
    newRecord["layer"] = "NGINX"
    newRecord["service"] = "nginx::nginx"
    newRecord["serviceInstance"] = "nginx-instance"
    newRecord["body"] = { text = { text = record.log } }
    newRecord["tags"] = { data = {{ key = "LOG_KIND", value = "NGINX_ACCESS_LOG"}}}
    return 1, timestamp, newRecord
end

function rewrite_error_log(tag, timestamp, record)
    local newRecord = {}
    newRecord["layer"] = "NGINX"
    newRecord["service"] = "nginx::nginx"
    newRecord["serviceInstance"] = "nginx-instance"
    newRecord["body"] = { text = { text = record.log } }
    newRecord["tags"] = { data = {{ key = "LOG_KIND", value = "NGINX_ERROR_LOG" }}}
    return 1, timestamp, newRecord
end

After starting fluent-it, we can see the collected log information in the Log tab of the monitoring panel：

Meaning of Metrics

Metric Name	Unit	Description	Data Source
HTTP Request Trend		The increment rate of HTTP requests	nginx-lua-prometheus
HTTP Latency	ms	The increment rate of the latency of HTTP requests	nginx-lua-prometheus
HTTP Bandwidth	KB	The increment rate of the bandwidth of HTTP requests	nginx-lua-prometheus
HTTP Connections		The avg number of the connections	nginx-lua-prometheus
HTTP Status Trend	%	The increment rate of the status of HTTP requests	nginx-lua-prometheus
HTTP Status 4xx Percent	%	The percentage of 4xx status of HTTP requests	nginx-lua-prometheus
HTTP Status 5xx Percent	%	The percentage of 4xx status of HTTP requests	nginx-lua-prometheus
Error Log Count		The count of log level of nginx error.log	fluent-bit

References

Zh: 使用 SkyWalking 监控 Nginx

Sat, 23 Dec 2023 00:00:00 +0000

背景介绍

在前面的 Blog 使用 LAL 收集并分析 Nginx access log 中，我们以 Nginx access log 为切入点，演示了 SkyWalking LAL 的日志分析能力。

为了实现对 Nginx 更全面的监控能力，我们在 SkyWalking 9.7 中引入了 Nginx 监控面板，本文将演示该监控面板的使用，并介绍相关指标的含义。

监控面板接入

Metric 定义与采集

由于使用了 nginx-lua-prometheus 来定义及暴露指标，我们需要为 Nginx 安装 lua_nginx_module，或者直接使用OpenResty。

下面的例子中，我们通过 nginx-lua-prometheus 定义了四个指标，并通过 ip:9145/metrics 暴露指标接口：

histogram: nginx_http_latency，监控 http 延时
gauge: nginx_http_connections，监控 http 连接数
counter: nginx_http_size_bytes，监控 http 请求和响应大小
counter: nginx_http_requests_total，监控 http 请求次数

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    lua_shared_dict prometheus_metrics 10M;
    # lua_package_path "/path/to/nginx-lua-prometheus/?.lua;;";

    init_worker_by_lua_block {
      prometheus = require("prometheus").init("prometheus_metrics")

      metric_bytes = prometheus:counter(
        "nginx_http_size_bytes", "Total size of HTTP", {"type", "route"})
      metric_requests = prometheus:counter(
        "nginx_http_requests_total", "Number of HTTP requests", {"status", "route"})
      metric_latency = prometheus:histogram(
        "nginx_http_latency", "HTTP request latency", {"route"})
      metric_connections = prometheus:gauge(
        "nginx_http_connections", "Number of HTTP connections", {"state"})
    }

    server {
        listen 8080;

        location /test {
          default_type application/json;
          return 200  '{"code": 200, "message": "success"}';

          log_by_lua_block {
            metric_bytes:inc(tonumber(ngx.var.request_length), {"request", "/test/**"})
            metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", "/test/**"})
            metric_requests:inc(1, {ngx.var.status, "/test/**"})
            metric_latency:observe(tonumber(ngx.var.request_time), {"/test/**"})
          }
        }
    }

    server {
      listen 9145;
      location /metrics {
        content_by_lua_block {
          metric_connections:set(ngx.var.connections_reading, {"reading"})
          metric_connections:set(ngx.var.connections_waiting, {"waiting"})
          metric_connections:set(ngx.var.connections_writing, {"writing"})
          prometheus:collect()
        }
      }
    }
}

上面的例子中，我们暴露了 route 级别的指标，你也可以根据监控粒度的需要，选择暴露 host 指标：

http {
  log_by_lua_block {
      metric_bytes:inc(tonumber(ngx.var.request_length), {"request", ngx.var.host})
      metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", ngx.var.host})
      metric_requests:inc(1, {ngx.var.status, ngx.var.host})
      metric_latency:observe(tonumber(ngx.var.request_time), {ngx.var.host})
  }
}

或者 upstream 指标：

upstream backend {
  server ip:port;
}

server {
  
  location /test_upstream {
  
    proxy_pass http://backend;
  
    log_by_lua_block {
      metric_bytes:inc(tonumber(ngx.var.request_length), {"request", "upstream/backend"})
      metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", "upstream/backend"})
      metric_requests:inc(1, {ngx.var.status, "upstream/backend"})
      metric_latency:observe(tonumber(ngx.var.request_time), {"upstream/backend"})
    }
  }
}

完成指标定义后，我们启动 nginx 和 opentelemetry-collector，将指标采集到 SkyWalking 后端进行分析和存储。

请确保job_name: 'nginx-monitoring'，否则上报的数据将被 SkyWalking 忽略。如果你有多个 Nginx 实例，你可以通过service及service_instance_id这两个 label 进行区分：

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'nginx-monitoring'
          scrape_interval: 5s
          metrics_path: "/metrics"
          static_configs:
            - targets: ['nginx:9145']
              labels:
                service: nginx
                service_instance_id: nginx-instance
processors:
  batch:

exporters:
  otlp:
    endpoint: oap:11800
    tls:
      insecure: true
service:
  pipelines:
    metrics:
      receivers:
        - prometheus
      processors:
        - batch
      exporters:
        - otlp

如果一切顺利，你将在 skywalking-ui 的网关菜单下看到 nginx 上报的指标数据：

Access & Error Log 采集

SkyWalking Nginx 监控提供了日志采集及错误日志统计功能，我们可以借助 fluent-bit 采集并上报 access log、error log 给 SkyWalking 分析存储。

下面 fluent-bit 配置定义了日志采集目录为/var/log/nginx/，access 和 error log 经过 rewrite_access_log 和 rewrite_error_log 处理后会通过 oap 12800 端口进行上报：

[SERVICE]
    Flush          5
    Daemon         Off
    Log_Level      warn
[INPUT]
    Name           tail
    Tag            access
    Path           /var/log/nginx/access.log
[INPUT]
    Name           tail
    Tag            error
    Path           /var/log/nginx/error.log
[FILTER]
    Name           lua
    Match          access
    Script         fluent-bit-script.lua
    Call           rewrite_access_log
[FILTER]
    Name           lua
    Match          error
    Script         fluent-bit-script.lua
    Call           rewrite_error_log
[OUTPUT]
    Name            stdout
    Match           *
    Format          json
[OUTPUT]
    Name            http
    Match           *
    Host            oap
    Port            12800
    URI             /v3/logs
    Format          json

在 fluent-bit-script.lua 中，我们通过 LOG_KIND 来区分 access log 和 error log。

为了能够关联上文采集的 metric，请确保 service 和 serviceInstance 值与上文中指标采集定义一致。

function rewrite_access_log(tag, timestamp, record)
    local newRecord = {}
    newRecord["layer"] = "NGINX"
    newRecord["service"] = "nginx::nginx"
    newRecord["serviceInstance"] = "nginx-instance"
    newRecord["body"] = { text = { text = record.log } }
    newRecord["tags"] = { data = {{ key = "LOG_KIND", value = "NGINX_ACCESS_LOG"}}}
    return 1, timestamp, newRecord
end

function rewrite_error_log(tag, timestamp, record)
    local newRecord = {}
    newRecord["layer"] = "NGINX"
    newRecord["service"] = "nginx::nginx"
    newRecord["serviceInstance"] = "nginx-instance"
    newRecord["body"] = { text = { text = record.log } }
    newRecord["tags"] = { data = {{ key = "LOG_KIND", value = "NGINX_ERROR_LOG" }}}
    return 1, timestamp, newRecord
end

启动 fluent-it 后，我们便可以在监控面板的 Log tab 看到采集到的日志信息：

面板指标含义

面板名称	单位	指标含义	数据源
HTTP Request Trend		每秒钟平均请求数	nginx-lua-prometheus
HTTP Latency	ms	平均响应延时	nginx-lua-prometheus
HTTP Bandwidth	KB	请求响应流量	nginx-lua-prometheus
HTTP Connections		nginx http 连接数	nginx-lua-prometheus
HTTP Status Trend	%	每分钟 http 状态码统计	nginx-lua-prometheus
HTTP Status 4xx Percent	%	4xx状态码比例	nginx-lua-prometheus
HTTP Status 5xx Percent	%	5xx状态码比例	nginx-lua-prometheus
Error Log Count		每分钟错误日志数统计	fluent-bit

参考文档

Zh: 使用 LAL 收集并分析 Nginx access log

Sun, 29 Oct 2023 00:00:00 +0000

背景介绍

Nginx access log 中包含了丰富的信息，例如：日志时间、状态码、响应时间、body 大小等。通过收集并分析 access log，我们可以实现对 Nginx 中接口状态的监控。

在本案例中，将由 fluent-bit 收集 access log，并通过 HTTP 将日志信息发送给 SkyWalking OAP Server 进行进一步的分析。

环境准备

实验需要的 Nginx 及 Fluent-bit 相关配置文件都被上传到了Github，有需要的读者可以自行 git clone 并通过 docker compose 启动，本文中将介绍配置文件中几个关键点。

Nginx日志格式配置

LAL 目前支持 JSON、YAML 及 REGEX 日志解析，为了方便获取到日志中的指标字段，我们将 Nginx 的日志格式定义为 JSON.

http {
    ...
    ...

    log_format  main  '{"remote_addr": "$remote_addr",'
            '"remote_user": "$remote_user",'
            '"request": "$request",'
            '"time": "$time_iso8601",'
            '"status": "$status",'
            '"request_time":"$request_time",'
            '"body_bytes_sent": "$body_bytes_sent",'
            '"http_referer": "$http_referer",'
            '"http_user_agent": "$http_user_agent",'
            '"http_x_forwarded_for": "$http_x_forwarded_for"}';

    access_log  /var/log/nginx/access.log  main;
    
    ...
    ...
}

Fluent bit Filter

我们通过 Fluent bit 的 lua filter 进行日志格式的改写，将其调整为 SkyWalking 所需要的格式，record的各个字段含义如下：

body：日志内容体
service：服务名称
serviceInstance：实例名称

function rewrite_body(tag, timestamp, record)
    local newRecord = {}
    newRecord["body"] = { json = { json = record.log } }
    newRecord["service"] = "nginx::nginx"
    newRecord["serviceInstance"] = "localhost"
    return 1, timestamp, newRecord
end

OAP 日志分析

LAL定义

在 filter 中，我们通过条件判断，只处理 service=nginx::nginx 的服务，其他服务依旧走默认逻辑：

第一步，使用 json 指令对日志进行解析，解析的结果会被存放到 parsed 字段中，通过 parsed 字段我们可以获取 json 日志中的字段信息。

第二步，使用 timestamp 指令解析 parsed.time 并将其赋值给日志的 timestamp 字段，这里的 time 就是access log json 中的 time。

第三步，使用 tag 指令给日志打上对应的标签，标签的值依然可以通过 parsed 字段获取。

第四步，使用 metrics 指令从日志中提取出指标信息，我们共提取了四个指标：

nginx_log_count：Nginx 每次请求都会生成一条 access log，该指标可以帮助我们统计 Nginx 当前的请求数。
nginx_request_time：access log 中会记录请求时间，该指标可以帮助我们统计上游接口的响应时长。
nginx_body_bytes_sent：body 大小指标可以帮助我们了解网关上的流量情况。
nginx_status_code：状态码指标可以实现对状态码的监控，如果出现异常上涨可以结合 alarm 进行告警。

rules:
  - name: default
    layer: GENERAL
    dsl: |
      filter {
        if (log.service == "nginx::nginx") {
          json {
            abortOnFailure true
          }
    
          extractor {
            timestamp parsed.time as String, "yyyy-MM-dd'T'HH:mm:ssXXX"
            tag status: parsed.status
            tag remote_addr: parsed.remote_addr
      
            metrics {
              timestamp log.timestamp as Long
              labels service: log.service, instance: log.serviceInstance
              name "nginx_log_count"
              value 1
            }
            metrics {
              timestamp log.timestamp as Long
              labels service: log.service, instance: log.serviceInstance
              name "nginx_request_time"
              value parsed.request_time as Double
            }
            metrics {
              timestamp log.timestamp as Long
              labels service: log.service, instance: log.serviceInstance
              name "nginx_body_bytes_sent"
              value parsed.body_bytes_sent as Long
            }
            metrics {
              timestamp log.timestamp as Long
              labels service: log.service, instance: log.serviceInstance, status: parsed.status
              name "nginx_status_code"
              value 1
            }
          }
        }
      
        sink {
        }
      }

经过 LAL 处理后，我们已经可以在日志面板看到日志信息了，接下来我们将对 LAL 中提取的指标进行进一步分析：

MAL定义

在 MAL 中，我们可以对上一步 LAL 中提取的指标进行进一步的分析聚合，下面的例子里：

nginx_log_count、nginx_request_time、nginx_status_code 使用 sum 聚合函数处理，并使用 SUM 方式 downsampling，

nginx_request_time 使用 avg 聚合函数求平均值，默认使用 AVG 方式 downsampling。

完成聚合分析后，SkyWalking Meter System 会完成对上述指标的持久化。

expSuffix: service(['service'], Layer.GENERAL)
metricPrefix: nginx
metricsRules:
  - name: cpm
    exp: nginx_log_count.sum(['service']).downsampling(SUM)
  - name: avg_request_time
    exp: nginx_request_time.avg(['service'])
  - name: body_bytes_sent_count
    exp: nginx_body_bytes_sent.sum(['service']).downsampling(SUM)
  - name: status_code_count
    exp: nginx_status_code.sum(['service','status']).downsampling(SUM)

最后，我们便可以来到 SkyWalking UI 页面新建 Nginx 仪表板，使用刚刚 MAL 中定义的指标信息创建 Nginx Dashboard（也可以通过上文提到仓库中的 dashboard.json 直接导入测试）：

参考文档

Blog: How to integrate skywalking-nginx-lua to Nginx?

Mon, 13 Dec 2021 00:00:00 +0000

We Can integrate Skywalking to Java Application by Java Agent TEC.， In typical application, the system runs Java Web applications at the backend of the load balancer, and the most commonly used load balancer is nginx. What should we do if we want to bring it under surveillance? Fortunately, skywalking has provided Nginx agent。 During the integration process, it is found that the examples on the official website only support openresty. For openresty, common modules such as luajit and Lua nginx module have been integrated. Adding skywalking related configurations according to the examples on the official website can take effect. However, when configured for nginx startup, many errors will be reported. We may not want to change a load balancer (nginx to openresty) in order to use skywalking. Therefore, we must solve the integration problem between skywalking and nginx.

Note: openresty is a high-performance web development platform based on nginx + Lua, which solves the short board that is not easy to program in nginx.

Based on Skywalking-8.7.0 and Nginx-1.20.1

Upgrade of nginx:

The agent plug-in of nginx is written based on Lua, so nginx needs to add support for Lua, Lua nginx module It just provides this function. The Lua nginx module depends on luajit Therefore, first we need to install luajit. In the environment, it is best to choose version 2.1.

For nginx, you need to compile the necessary modules yourself. It depends on the following two modules:

lua-nginx-module The version is lua-nginx-module-0.10.21rc1

ngx_devel_kit The version using ngx_devel_kit-0.3.1

Compile nginx parameters


configure arguments: --add-module=/path/to/ngx_devel_kit-0.3.1 --add-module=/path/to/lua-nginx-module-0.10.21rc1 --with-ld-opt=-Wl,-rpath,/usr/local/LuaJIT/lib

The following is for skywalking-nginx-lua-0.3.0 and 0.3.0+ are described separately.

skywalking-nginx-lua-0.3.0

After testing, skywalking-nginx-lua-0.3.0 requires the following Lua related modules


lua-resty-core https://github.com/openresty/lua-resty-core

lua-resty-lrucache https://github.com/openresty/lua-resty-lrucache

lua-cjson https://github.com/openresty/lua-cjson

The dependent Lua modules are as follows:


lua_package_path "/path/to/lua-resty-core/lua-resty-core-master/lib/?.lua;/path/to/lua-resty-lrucache-0.11/lib/?.lua;/path/to/skywalking-nginx-lua-0.3.0/lib/?.lua;;";

In the process of make & & make install, Lua cjson needs to pay attention to:

Modify a path in makefile

LUA_INCLUDE_DIR ?= /usr/local/LuaJIT/include/luajit-2.0

Reference: https://blog.csdn.net/ymeputer/article/details/50146143

skywalking-nginx-lua-0.3.0+

For skywalking-nginx-lua-0.3.0+, tablepool support needs to be added, but it seems that cjson is not required


lua-resty-core https://github.com/openresty/lua-resty-core

lua-resty-lrucache https://github.com/openresty/lua-resty-lrucache

lua-tablepool https://github.com/openresty/lua-tablepool


lua_ package_ path "/path/to/lua-resty-core/lua-resty-core-master/lib/?.lua;/path/to/lua-resty-lrucache-0.11/lib/?.lua;/path/to/lua-tablepool-master/lib/?.lua;/path/to/skywalking-nginx-lua-master/lib/?.lua;;";

tablepool introduces two APIs according to its official documents table new and table. Clear requires luajit2.1, there is a paragraph in the skywalking-nginx-lua document that says you can use ‘require (“skywalking. Util”) disable_ Tablepool() ` disable tablepool

When you start nginx, you will be prompted to install openresty’s own [luajit version]（ https://github.com/openresty/luajit2 )


detected a LuaJIT version which is not OpenResty's; many optimizations will be disabled and performance will be compromised (see https://github.com/openresty/luajit2 for OpenResty's LuaJIT or, even better, consider using the OpenResty releases from https://openresty.org/en/download.html )

here is successful configuration:

     http {
    lua_package_path "/path/to/lua-resty-core/lua-resty-core-master/lib/?.lua;/path/to/lua-resty-lrucache-0.11/lib/?.lua;/path/to/lua-tablepool-master/lib/?.lua;/path/to/skywalking-nginx-lua-master/lib/?.lua;;";

    # Buffer represents the register inform and the queue of the finished segment
    lua_shared_dict tracing_buffer 100m;

    # Init is the timer setter and keeper
    # Setup an infinite loop timer to do register and trace report.
    init_worker_by_lua_block {
        local metadata_buffer = ngx.shared.tracing_buffer

        -- Set service name
        metadata_buffer:set('serviceName', 'User Service Name')
        -- Instance means the number of Nginx deployment, does not mean the worker instances
        metadata_buffer:set('serviceInstanceName', 'User Service Instance Name')
        -- type 'boolean', mark the entrySpan include host/domain
        metadata_buffer:set('includeHostInEntrySpan', false)

        -- set random seed
        require("skywalking.util").set_randomseed()
        require("skywalking.client"):startBackendTimer("http://127.0.0.1:12800")

        -- If there is a bug of this `tablepool` implementation, we can
        -- disable it in this way
        -- require("skywalking.util").disable_tablepool()

        skywalking_tracer = require("skywalking.tracer")
    }

    server {
        listen 8090;
        
        location /ingress {
            default_type text/html;

            rewrite_by_lua_block {
                ------------------------------------------------------
                -- NOTICE, this should be changed manually
                -- This variable represents the upstream logic address
                -- Please set them as service logic name or DNS name
                --
                -- Currently, we can not have the upstream real network address
                ------------------------------------------------------
                skywalking_tracer:start("upstream service")
                -- If you want correlation custom data to the downstream service
                -- skywalking_tracer:start("upstream service", {custom = "custom_value"})
            }

            -- Target upstream service
            proxy_pass http://127.0.0.1:8080/backend;

            body_filter_by_lua_block {
                if ngx.arg[2] then
                    skywalking_tracer:finish()
                end
            }

            log_by_lua_block {
                skywalking_tracer:prepareForReport()
            }
        }
    }
}

Original post：https://www.cnblogs.com/kebibuluan/p/14440228.html

Apache SkyWalking – Nginx

Zh: SkyWalking 如何支持 ZIO 等 Scala Effect Runtime

背景介绍

ZIO Trace

分析

默认线程池 defaultExecutor

阻塞线程池 blockingExecutor

串连 Fiber 上下文

说明

Blog: Monitoring Nginx with SkyWalking

Background

Setup Monitoring Dashboard

Metric Define and Collection

Access & Error Log Collection

Meaning of Metrics

References

Zh: 使用 SkyWalking 监控 Nginx

背景介绍

监控面板接入

Metric 定义与采集

Access & Error Log 采集

面板指标含义

参考文档

Zh: 使用 LAL 收集并分析 Nginx access log

背景介绍

环境准备

Nginx日志格式配置

Fluent bit Filter

OAP 日志分析

LAL定义

MAL定义

参考文档

Blog: How to integrate skywalking-nginx-lua to Nginx?

Upgrade of nginx:

skywalking-nginx-lua-0.3.0

skywalking-nginx-lua-0.3.0+

默认线程池 `defaultExecutor`

阻塞线程池 `blockingExecutor`