Apache SkyWalking – Agent

Blog: SkyWalking Ruby Quick Start and Principle Introduction

Thu, 06 Mar 2025 00:00:00 +0000

Background

Ruby is a dynamic, object-oriented programming language with concise and elegant syntax, supporting multiple programming paradigms, including object-oriented, functional, and metaprogramming. Leveraging its powerful metaprogramming capabilities, Ruby allows modifying the behavior of classes and objects at runtime. SkyWalking provides a Ruby gem to facilitate integration with Ruby projects, and this gem supports many out-of-the-box frameworks and gems.

This article is based on skywalking-ruby-v0.1. We will guide you on how to quickly integrate the skywalking-ruby project into Ruby projects and briefly introduce the implementation principle of SkyWalking Ruby’s auto-instrumentation plugins using redis-rb as an example.

The demonstration includes the following steps:

Deploy SkyWalking: This involves setting up the SkyWalking backend and UI programs to enable you to see the final results.
Integrate SkyWalking into Different Ruby Projects: This section explains how to integrate SkyWalking into different Ruby projects.
Application Deployment: You will export environment variables and deploy the application to facilitate communication between your service and the SkyWalking backend.
Visualization on SkyWalking UI: Finally, you will send requests and observe the results in the SkyWalking UI.

Deploy SkyWalking

Please download the SkyWalking APM program from the official SkyWalking website, and then you can start all the required services using the quick start script.

Next, you can access the address http://localhost:8080/. At this point, since no applications have been deployed, you will not see any data.

Integrate SkyWalking into Different Ruby Projects It is recommended to use Bundler to install and manage SkyWalking dependencies. Simply declare it in the Gemfile and run bundle install to complete the installation.

# Gemfile
source "https://rubygems.org"

gem "skywalking"

Integration in Rails Projects

For Rails projects, it is recommended to use the following command to automatically generate the configuration file:

bundle exec rails generate skywalking:start

This command will automatically generate a skywalking.rb file in the config/initializers directory, where you can configure the startup parameters.

Integration in Sinatra Projects

For Sinatra projects, you need to manually call Skywalking.start when the application starts. For example:

require 'sinatra'
require 'skywalking'

Skywalking.start

get '/sw' do
  "Hello SkyWalking!"
end

In the Gemfile, place skywalking after sinatra and use Bundler.require during initialization, or call require 'skywalking' after the sinatra gem is loaded. Note that the skywalking gem needs to be placed after other gems (such as redis, elasticsearch).

Application Deployment

Before starting the application deployment, you can change the service name of the current application in SkyWalking through environment variables. You can also modify its configuration, such as the server-side address. For more details, please refer to the documentation.

Here, we will change the current service name to sw-ruby.

Next, you can start the application. Here is an example using sinatra:

export SW_AGENT_SERVICE_NAME=sw-ruby

ruby sinatra.rb

Visualization on SkyWalking UI

Now, send requests to the application and observe the results in the SkyWalking UI.

After a few seconds, revisit the SkyWalking UI at http://localhost:8080. You will be able to see the deployed demo service on the homepage.

Additionally, on the tracing page, you can see the request you just sent.

Plugin Implementation Mechanism

To understand the implementation mechanism of Ruby Agent’s auto-instrumentation plugins, it is essential to understand the concept of the ancestor chain in Ruby. The ancestor chain is an ordered list, and in Ruby, each class or module has an ancestor chain that includes all its parent classes and mixin modules (modules mixed in via include, prepend, or extend). When Ruby looks up a method, it searches in the order of the ancestor chain until it finds the target method or throws a NoMethodError.

class User
end

We have defined a User class, and its ancestor chain is as shown in the following figure:

Next, mix in a module using the prepend method:

module Dapper
  def brave
    "Hello from brave"
  end
end

class User
  prepend Dapper
end

p User.new.brave # => "Hello from brave"

prepend will insert at position 1 in the above figure. Ruby first looks for the brave method in the Dapper module, finds it, and calls it. If the brave method is not found in Dapper, Ruby continues to search in the User class. If it is not found in the User class, Ruby continues to search in Object, and so on.

Based on this mechanism, let’s briefly introduce how we instrument the redis-rb method. The following code is the target method to be instrumented:

# lib/redis/client.rb
class Redis
  class Client < ::RedisClient
    def call_v(command, &block)
      super(command, &block)
    rescue ::RedisClient::Error => error
      Client.translate_error!(error)
    end
  end
end

Below is the core code for instrumentation:

module Skywalking
  module Plugins
    class Redis5 < PluginsManager::SWPlugin
      module Redis5Intercept
        def call_v(args, &block)
          operation = args[0] rescue "UNKNOWN"
          return super if operation == :auth

          Tracing::ContextManager.new_exit_span(
            operation: "Redis/#{operation.upcase}"
          ) do |span|
            # Omitted handling of span 
            super(args, &block) # Call the original method
          end
        end
      end

      def install
        ::Redis::Client.prepend Redis5Intercept
      end
    end
  end
end

Here, we define a Redis5Intercept module and prepend it to ::Redis::Client. According to Ruby’s method lookup mechanism, when the call_v method of Redis::Client is called, Ruby will first execute the call_v method in Redis5Intercept. The order of the ancestor chain is as follows:

Redis5Intercept -> Redis::Client -> ... (other parent classes and modules)

At the same time, in the call_v method of Redis5Intercept, super(args, &block) will find the next method with the same name along the ancestor chain, which in this case is the original call_v method in Redis::Client, while passing the original arguments and block.

Conclusion

This article explained the integration methods of SkyWalking Ruby in Ruby projects and briefly introduced the implementation mechanism of SkyWalking Ruby’s auto-instrumentation plugins.

Currently, the Ruby auto-instrumentation is in the early stages of development. In the future, we will continue to expand the functionality of SkyWalking Ruby and add support for more plugins. So, stay tuned!

Zh: SkyWalking Ruby 快速开始与原理介绍

Thu, 06 Mar 2025 00:00:00 +0000

背景

Ruby 是一种动态、面向对象的编程语言，它的语法简洁优雅，支持多种编程范式，包括面向对象、函数式和元编程。其中依靠强大的元编程能力，Ruby 允许在运行时修改类和对象的行为。 SkyWalking 提供了 Ruby gem，方便 Ruby 项目集成, 该 gem 支持许多开箱即用的框架和 gem。

本文基于 skywalking-ruby-v0.1，我们将指导你如何快速将 skywalking-ruby 项目集成到 Ruby 项目中，并以 redis-rb 为例，简要地介绍 SkyWalking Ruby 对插件自动探针的实现原理。

演示部分包括以下步骤：

部署 SkyWalking：这涉及设置 SkyWalking 后端和 UI 程序，使你能够看到最终效果。
为不同 Ruby 项目集成 skywalking：这里介绍了不同的 Ruby 项目如何集成 skywalking。
应用部署：你将导出环境变量并部署应用程序，以促进你的服务与 SkyWalking 后端之间的通信。
在 SkyWalking UI 上可视化：最后，你将发送请求并在 SkyWalking UI 中观察效果。

部署 SkyWalking

请从官方 SkyWalking 网站下载 SkyWalking APM 程序，然后可以根据快速启动脚本启动所有所需服务。

接下来，你可以访问地址 http://localhost:8080/ 。此时，由于尚未部署任何应用程序，因此你将看不到任何数据。

为不同 Ruby 项目集成 SkyWalking

推荐使用 Bundler 来安装和管理 skywalking 的依赖。只需在 Gemfile 中声明，然后运行 bundle install 即可完成安装。

# Gemfile
source "https://rubygems.org"

gem "skywalking"

在 Rails 项目中集成

对于 Rails 项目，推荐使用以下命令自动生成配置文件：

bundle exec rails generate skywalking:start

该命令会在 config/initializers 目录下自动生成 skywalking.rb 文件，你可以在其中配置启动参数。

在 Sinatra 项目中集成

对于 Sinatra 项目，你需要手动在应用启动时调用 Skywalking.start。例如：

require 'sinatra'
require 'skywalking'

Skywalking.start

get '/sw' do
  "Hello SkyWalking!"
end

在 Gemfile 中，将 skywalking 放在 sinatra 之后，并在初始化时使用 Bundler.require，或者在 sinatra gem 加载后调用 require ‘skywalking’。注意，skywalking gem 需要位于其他 gem（如 redis、elasticsearch）之后。

应用部署

在开始部署应用程序之前，你可以通过环境变量更改 SkyWalking 中当前应用程序的服务名称。你还可以更改其配置，例如服务器端的地址。有关详细信息，请参阅文档。

在这里，我们将当前服务的名称更改为 sw-ruby。

接下来，你可以启动应用程序，这里以 sinatra 作为示例：

export SW_AGENT_SERVICE_NAME=sw-ruby

ruby sinatra.rb

在 SkyWalking UI 上可视化

现在，向应用程序发送请求并在 SkyWalking UI 中观察结果。

几秒钟后，重新访问 http://localhost:8080 的 SkyWalking UI。能够在主页上看到部署的 demo 服务。

此外，在追踪页面上，可以看到刚刚发送的请求。

插件实现机制

要了解 Ruby Agent 对插件自动探针的实现机制，首先要了解 Ruby 中祖先链的概念。祖先链是一个有序的列表，在 Ruby 中，每个类或模块都有一个祖先链，它包含了一个类或模块的所有父类以及 mixin 模块（通过 include、prepend 或 extend 混入的模块）。 Ruby 在查找方法时，会按照祖先链的顺序依次查找，直到找到目标方法或抛出 NoMethodError。

class User
end

我们定义了一个 User 类，那么它的祖先链是如下图：

接下来用 prepend 方法混入一个模块：

module Dapper
  def brave
    "Hello from brave"
  end
end

class User
  prepend Dapper
end

p User.new.brave # => "Hello from brave"

prepend 会在上图 1 处进行插入，Ruby 首先在 Dapper 模块中查找 brave 方法，找到并调用，如果 Dapper 中没有 brave 方法， Ruby 会继续查找 User 类。如果 User 类中也没有，Ruby 会继续查找 Object，依此类推。

根据这样的机制，简单介绍下我们如何对 redis-rb 进行方法插桩，下面代码是要进行插桩的目标方法：

# lib/redis/client.rb
class Redis
  class Client < ::RedisClient
    def call_v(command, &block)
     super(command, &block)
    rescue ::RedisClient::Error => error
     Client.translate_error!(error)
    end
  end
end

下面是进行插桩的核心代码：

module Skywalking
  module Plugins
    class Redis5 < PluginsManager::SWPlugin
      module Redis5Intercept
        def call_v(args, &block)
          operation = args[0] rescue "UNKNOWN"
          return super if operation == :auth

          Tracing::ContextManager.new_exit_span(
            operation: "Redis/#{operation.upcase}"
          ) do |span|
            # 省略对 span 的处理 
            super(args, &block) # 调用原方法
          end
        end
      end

      def install
        ::Redis::Client.prepend Redis5Intercept
      end
    end
  end
end

这里我们定义了一个 Redis5Intercept 模块，并将其作为 ::Redis::Client 的前置模块，根据 Ruby 方法查找机制，当 Redis::Client 的 call_v 方法被调用时，Ruby 会首先会执行 Redis5Intercept 中的 call_v 方法，这里祖先链的顺序如下：

Redis5Intercept -> Redis::Client -> ...（其他父类和模块）

同时在 Redis5Intercept 中的 call_v 方法中，super(args, &block) 会沿着祖先链找到下一个同名方法，在这里也就是 Redis::Client 中的原始 call_v 方法，同时传递原始的参数和代码块。

总结

本文讲述了 Skywalking Ruby 在 Ruby 项目中的集成方法，并简要介绍了 SkyWalking Ruby 对插件自动探针的实现机制。

目前 Ruby 探针处于早期的开发阶段，未来我们将继续扩展 SkyWalking Ruby 的功能，添加更多插件支持。所以，请继续关注！

Zh: CommunityOverCode Conference 2023 Asia

Tue, 03 Oct 2023 00:00:00 +0000

CommunityOverCode (原 ApacheCon) 是 Apache 软件基金会（ASF）的官方全球系列大会。自 1998 年以来–在 ASF 成立之前 – ApacheCon 已经吸引了各个层次的参与者，在 300 多个 Apache 项目及其不同的社区中探索 “明天的技术”。CommunityOverCode 通过动手实作、主题演讲、实际案例研究、培训、黑客松活动等方式，展示 Apache 项目的最新发展和新兴创新。

CommunityOverCode 展示了无处不在的 Apache 项目的最新突破和 Apache 孵化器中即将到来的创新，以及开源开发和以 Apache 之道领导社区驱动的项目。与会者可以了解到独立于商业利益、企业偏见或推销话术之外的核心开源技术。

SkyWalking的Golang自动探针实践刘晗

分布式追踪技术在可观测领域尤为重要，促使各个语言的追踪探针的易用性获得了更多的关注。目前在golang语言探针方面大多为手动埋点探针，接入流程过于复杂，而且局限性很强。本次讨论的重点着重于简化golang语言探针的接入方式，创新性的使用了自动埋点技术，并且突破了很多框架中对于上下文信息的依赖限制。

B站视频地址

BanyanDB一个高扩展性的分布式追踪数据库高洪涛

追踪数据是一种用于分析微服务系统性能和故障的重要数据源，它记录了系统中每个请求的调用链路和相关指标。随着微服务系统的规模和复杂度的增长，追踪数据的量级也呈指数级增长，给追踪数据的存储和查询带来了巨大的挑战。传统的关系型数据库或者时序数据库往往难以满足追踪数据的高效存储和灵活查询的需求。 BanyanDB是一个专为追踪数据而设计的分布式数据库，它具有高扩展性、高性能、高可用性和高灵活性的特点。BanyanDB采用了基于时间序列的分片策略，将追踪数据按照时间范围划分为多个分片，每个分片可以独立地进行存储、复制和负载均衡。BanyanDB还支持多维索引，可以根据不同的维度对追踪数据进行快速过滤和聚合。在本次演讲中，我们将介绍BanyanDB的设计思想、架构和实现细节，以及它在实际场景中的应用和效果。我们也将展示BanyanDB与其他数据库的对比和优势，以及它未来的发展方向和计划。

B站视频地址

Zh: 将 Apache SkyWalking 与 Arthas 集成

Sun, 17 Sep 2023 00:00:00 +0000

背景介绍

Arthas 是一款常用的 Java 诊断工具，我们可以在 SkyWalking 监控到服务异常后，通过 Arthas 进一步分析和诊断以快速定位问题。

在 Arthas 实际使用中，通常由开发人员拷贝或者下载安装包到服务对应的VM或者容器中，attach 到对应的 Java 进程进行问题排查。这一过程不可避免的会造成服务器敏感运维信息的扩散，而且在分秒必争的问题排查过程中，这些繁琐的操作无疑会浪费大量时间。

SkyWalking Java Agent 伴随 Java 服务一起启动，并定期上报服务、实例信息给OAP Server。我们可以借助 SkyWalking Java Agent 的插件化能力，开发一个 Arthas 控制插件，由该插件管理 Arthas 运行生命周期，通过页面化的方式，完成Arthas的启动与停止。最终实现效果可以参考下图：

要完成上述功能，我们需要实现以下几个关键点：

开发 agent arthas-control-plugin，执行 arthas 的启动与停止命令
开发 oap arthas-controller-module ，下发控制命令给 arthas agent plugin
定制 skywalking-ui, 连接 arthas-tunnel-server，发送 arthas 命令并获取执行结果

以上各个模块之间的交互流程如下图所示：

connect

disconnect

本文涉及的所有代码均已发布在 github skywalking-x-arthas 上，如有需要，大家可以自行下载代码测试。文章后半部分将主要介绍代码逻辑及其中包含的SkyWalking扩展点。

agent arthas-control-plugin

首先在 skywalking-java/apm-sniffer/apm-sdk-plugin 下创建一个 arthas-control-plugin，该模块在打包后会成为 skywalking-agent/plugins 下的一个插件，其目录结构如下：

arthas-control-plugin/
├── pom.xml
└── src
    └── main
        ├── java
        │   └── org
        │       └── apache
        │           └── skywalking
        │               └── apm
        │                   └── plugin
        │                       └── arthas
        │                           ├── config
        │                           │   └── ArthasConfig.java    # 模块配置
        │                           ├── service
        │                           │   └── CommandListener.java # boot service，监听 oap command
        │                           └── util
        │                               ├── ArthasCtl.java       # 控制 arthas 的启动与停止
        │                               └── ProcessUtils.java
        ├── proto
        │   └── ArthasCommandService.proto                       # 与oap server通信的 grpc 协议定义
        └── resources
            └── META-INF
                └── services                                     # boot service spi service
                    └── org.apache.skywalking.apm.agent.core.boot.BootService

16 directories, 7 files

在 ArthasConfig.java 中，我们定义了以下配置，这些参数将在 arthas 启动时传递。

以下的配置可以通过 agent.config 文件、system prop、env variable指定。关于 skywalking-agent 配置的初始化的具体流程，大家可以参考 SnifferConfigInitializer 。

public class ArthasConfig {
    public static class Plugin {
        @PluginConfig(root = ArthasConfig.class)
        public static class Arthas {
            // arthas 目录
            public static String ARTHAS_HOME;
            // arthas 启动时连接的tunnel server
            public static String TUNNEL_SERVER;
            // arthas 会话超时时间
            public static Long SESSION_TIMEOUT;
            // 禁用的 arthas command
            public static String DISABLED_COMMANDS;
        }
    }
}

接着，我们看下 CommandListener.java 的实现，CommandListener 实现了 BootService 接口，并通过 resources/META-INF/services 下的文件暴露给 ServiceLoader。

BootService 的定义如下，共有prepare()、boot()、onComplete()、shutdown()几个方法，这几个方法分别对应插件生命周期的不同阶段。

public interface BootService {
    void prepare() throws Throwable;

    void boot() throws Throwable;

    void onComplete() throws Throwable;

    void shutdown() throws Throwable;

    default int priority() {
        return 0;
    }
}

在 ServiceManager 类的 boot() 方法中，定义了BootService 的 load 与启动流程，该方法由SkyWalkingAgent 的 premain 调用，在主程序运行前完成初始化与启动：

public enum ServiceManager {
    INSTANCE;
    ...
    ...
    public void boot() {
        bootedServices = loadAllServices();

        prepare();
        startup();
        onComplete();
    }
    ...
    ...
}

回到我们 CommandListener 的 boot 方法，该方法在 agent 启动之初定义了一个定时任务，这个定时任务会轮询 oap ，查询是否需要启动或者停止arthas:

public class CommandListener implements BootService, GRPCChannelListener {
    ...
    ...
    @Override
    public void boot() throws Throwable {
        getCommandFuture = Executors.newSingleThreadScheduledExecutor(
            new DefaultNamedThreadFactory("CommandListener")
        ).scheduleWithFixedDelay(
            new RunnableWithExceptionProtection(
                this::getCommand,
                t -> LOGGER.error("get arthas command error.", t)
            ), 0, 2, TimeUnit.SECONDS
        );
    }
    ...
    ...
}

getCommand方法中定义了start、stop的处理逻辑，分别对应页面上的 connect 和 disconnect 操作。这两个 command 有分别转给 ArthasCtl 的 startArthas 和 stopArthas 两个方法处理，用来控制 arthas 的启停。

在 startArthas 方法中，启动arthas-core.jar 并使用 skywalking-agent 的 serviceName 和 instanceName 注册连接至配置文件中指定的arthas-tunnel-server。

ArthasCtl 逻辑参考自 Arthas 的 BootStrap.java ，由于不是本篇文章的重点，这里不再赘述，感兴趣的小伙伴可以自行查看。

switch (commandResponse.getCommand()) {
    case START:
        if (alreadyAttached()) {
            LOGGER.warn("arthas already attached, no need start again");
            return;
        }
        try {
            arthasTelnetPort = SocketUtils.findAvailableTcpPort();
            ArthasCtl.startArthas(PidUtils.currentLongPid(), arthasTelnetPort);
        } catch (Exception e) {
            LOGGER.info("error when start arthas", e);
        }
        break;
    case STOP:
        if (!alreadyAttached()) {
            LOGGER.warn("no arthas attached, no need to stop");
            return;
        }
        try {
            ArthasCtl.stopArthas(arthasTelnetPort);
            arthasTelnetPort = null;
        } catch (Exception e) {
            LOGGER.info("error when stop arthas", e);
        }
        break;
}

看完 arthas 的启动与停止控制逻辑，我们回到 CommandListener 的 statusChanged 方法，由于要和 oap 通信，这里我们按照惯例监听 grpc channel 的状态，只有状态正常时才会执行上面的getCommand轮询。

public class CommandListener implements BootService, GRPCChannelListener {
    ...
    ...
    @Override
    public void statusChanged(final GRPCChannelStatus status) {
        if (GRPCChannelStatus.CONNECTED.equals(status)) {
            Object channel = ServiceManager.INSTANCE.findService(GRPCChannelManager.class).getChannel();
            // DO NOT REMOVE Channel CAST, or it will throw `incompatible types: org.apache.skywalking.apm.dependencies.io.grpc.Channel
            // cannot be converted to io.grpc.Channel` exception when compile due to agent core's shade of grpc dependencies.
            commandServiceBlockingStub = ArthasCommandServiceGrpc.newBlockingStub((Channel) channel);
        } else {
            commandServiceBlockingStub = null;
        }
        this.status = status;
    }
    ...
    ...
}

上面的代码，细心的小伙伴可能会发现，getChannel() 的返回值被向上转型成了 Object, 而在下面的 newBlockingStub 方法中，又强制转成了 Channel。

看似有点多此一举，其实不然，我们将这里的转型去掉,尝试编译就会收到下面的错误：

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:compile (default-compile) on project arthas-control-plugin: Compilation failure
[ERROR] .../CommandListener.java:[59,103] 不兼容的类型: org.apache.skywalking.apm.dependencies.io.grpc.Channel无法转换为io.grpc.Channel

上面的错误提示 ServiceManager.INSTANCE.findService(GRPCChannelManager.class).getChannel() 的返回值类型是 org.apache.skywalking.apm.dependencies.io.grpc.Channel，无法被赋值给 io.grpc.Channel 引用。

我们查看GRPCChannelManager的getChannel()方法代码会发现，方法定义的返回值明明是 io.grpc.Channel，为什么编译时会报上面的错误？

其实这是skywalking-agent的一个小魔法，由于 agent-core 最终会被打包进 skywalking-agent.jar，启动时由系统类装载器（或者其他父级类装载器）直接装载，为了防止所依赖的类库和被监控服务的类发生版本冲突，agent 核心代码在打包时使用了maven-shade-plugin, 该插件会在 maven package 阶段改变 grpc 依赖的包名，我们在源代码里看到的是 io.grpc.Channel，其实在真正运行时已经被改成了 org.apache.skywalking.apm.dependencies.io.grpc.Channel，这便可解释上面编译报错的原因。

除了grpc以外，其他一些 well-known 的 dependency 也会进行 shade 操作，详情大家可以参考 apm-agent-core pom.xml ：

<plugin>
    <artifactId>maven-shade-plugin</artifactId>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                ...
                ...
                <relocations>
                    <relocation>
                        <pattern>${shade.com.google.source}</pattern>
                        <shadedPattern>${shade.com.google.target}</shadedPattern>
                    </relocation>
                    <relocation>
                        <pattern>${shade.io.grpc.source}</pattern>
                        <shadedPattern>${shade.io.grpc.target}</shadedPattern>
                    </relocation>
                    <relocation>
                        <pattern>${shade.io.netty.source}</pattern>
                        <shadedPattern>${shade.io.netty.target}</shadedPattern>
                    </relocation>
                    <relocation>
                        <pattern>${shade.io.opencensus.source}</pattern>
                        <shadedPattern>${shade.io.opencensus.target}</shadedPattern>
                    </relocation>
                    <relocation>
                        <pattern>${shade.io.perfmark.source}</pattern>
                        <shadedPattern>${shade.io.perfmark.target}</shadedPattern>
                    </relocation>
                    <relocation>
                        <pattern>${shade.org.slf4j.source}</pattern>
                        <shadedPattern>${shade.org.slf4j.target}</shadedPattern>
                    </relocation>
                </relocations>
                ...
                ...
            </configuration>
        </execution>
    </executions>
</plugin>

除了上面的注意点以外，我们来看一下另一个场景，假设我们需要在 agent plugin 的 interceptor 中使用 plugin 中定义的 BootService 会发生什么？

我们回到 BootService 的加载逻辑，为了加载到 plugin 中定义的BootService，ServiceLoader 指定了类装载器为AgentClassLoader.getDefault()，（这行代码历史非常悠久，可以追溯到2018年：Allow use SkyWalking plugin to override service in Agent core. #1111 ），由此可见，plugin 中定义的 BootService 的 classloader 是 AgentClassLoader.getDefault()：

void load(List<BootService> allServices) {
    for (final BootService bootService : ServiceLoader.load(BootService.class, AgentClassLoader.getDefault())) {
        allServices.add(bootService);
    }
}

再来看下 interceptor 的加载逻辑，InterceptorInstanceLoader.java 的 load 方法规定了如果父加载器相同，plugin 中的 interceptor 将使用一个新创建的 AgentClassLoader （在绝大部分简单场景中，plugin 的 interceptor 都由同一个 AgentClassLoader 加载）：

public static <T> T load(String className,
    ClassLoader targetClassLoader) throws IllegalAccessException, InstantiationException, ClassNotFoundException, AgentPackageNotFoundException {
    ...
    ...
    pluginLoader = EXTEND_PLUGIN_CLASSLOADERS.get(targetClassLoader);
    if (pluginLoader == null) {
        pluginLoader = new AgentClassLoader(targetClassLoader);
        EXTEND_PLUGIN_CLASSLOADERS.put(targetClassLoader, pluginLoader);
    }
    ...
    ...
}

按照类装载器的委派机制，interceptor 中如果用到了 BootService，也会由当前的类的装载器去装载。所以 ServiceManager 中装载的 BootService 和 interceptor 装载的 BootService 并不是同一个（一个 class 文件被不同的 classloader 装载了两次），如果在 interceptor 中调用 BootService 方法，同样会发生 cast 异常。由此可见，目前的实现并不支持我们在interceptor中直接调用 plugin 中 BootService 的方法，如果需要调用，只能将 BootService 放到 agent-core 中，由更高级别的类装载器优先装载。

这其实并不是 skywalking-agent 的问题，skywalking agent plugin 专注于自己的应用场景，只需要关注 trace、meter 以及默认 BootService 的覆盖就可以了。只是我们如果有扩展 skywalking-agent 的需求，要对其类装载机制做到心中有数，否则可能会出现一些意想不到的问题。

oap arthas-controller-module

看完 agent-plugin 的实现，我们再来看看 oap 部分的修改，oap 同样是模块化的设计，我们可以很轻松的增加一个新的模块，在 /oap-server/ 目录下新建 arthas-controller 子模块：

arthas-controller/
├── pom.xml
└── src
    └── main
        ├── java
        │   └── org
        │       └── apache
        │           └── skywalking
        │               └── oap
        │                   └── arthas
        │                       ├── ArthasControllerModule.java      # 模块定义
        │                       ├── ArthasControllerProvider.java    # 模块逻辑实现者
        │                       ├── CommandQueue.java
        │                       └── handler
        │                           ├── CommandGrpcHandler.java      # grpc handler,供 plugin 通信使用
        │                           └── CommandRestHandler.java      # http handler,供 skywalking-ui 通信使用
        ├── proto
        │   └── ArthasCommandService.proto
        └── resources
            └── META-INF
                └── services                                         # 模块及模块实现的 spi service
                    ├── org.apache.skywalking.oap.server.library.module.ModuleDefine
                    └── org.apache.skywalking.oap.server.library.module.ModuleProvider

模块的定义非常简单，只包含一个模块名，由于我们新增的模块并不需要暴露service给其他模块调用，services 我们返回一个空数组

public class ArthasControllerModule extends ModuleDefine {

    public static final String NAME = "arthas-controller";

    public ArthasControllerModule() {
        super(NAME);
    }

    @Override
    public Class<?>[] services() {
        return new Class[0];
    }
}

接着是模块实现者，实现者取名为 default，module 指定该 provider 所属模块，由于没有模块的自定义配置，newConfigCreator 我们返回null即可。 start 方法分别向 CoreModule 的 grpc 服务和 http 服务注册了两个 handler，grpc 服务和 http 服务就是我们熟知的 11800 和 12800 端口：

public class ArthasControllerProvider extends ModuleProvider {

    @Override
    public String name() {
        return "default";
    }

    @Override
    public Class<? extends ModuleDefine> module() {
        return ArthasControllerModule.class;
    }

    @Override
    public ConfigCreator<?> newConfigCreator() {
        return null;
    }

    @Override
    public void prepare() throws ServiceNotProvidedException {

    }

    @Override
    public void start() throws ServiceNotProvidedException, ModuleStartException {
        // grpc service for agent
        GRPCHandlerRegister grpcService = getManager().find(CoreModule.NAME)
                                                      .provider()
                                                      .getService(GRPCHandlerRegister.class);
        grpcService.addHandler(
            new CommandGrpcHandler()
        );

        // rest service for ui
        HTTPHandlerRegister restService = getManager().find(CoreModule.NAME)
                                                      .provider()
                                                      .getService(HTTPHandlerRegister.class);
        restService.addHandler(
            new CommandRestHandler(),
            Collections.singletonList(HttpMethod.POST)
        );
    }

    @Override
    public void notifyAfterCompleted() throws ServiceNotProvidedException {

    }

    @Override
    public String[] requiredModules() {
        return new String[0];
    }
}

最后在配置文件中注册本模块及模块实现者，下面的配置表示 arthas-controller 这个 module 由 default provider 提供实现：

arthas-controller:
  selector: default
  default:

CommandGrpcHandler 和 CommandHttpHandler 的逻辑非常简单，CommandHttpHandler 定义了 connect 和 disconnect 接口，收到请求后会放到一个 Queue 中供 CommandGrpcHandler 消费，Queue 的实现如下，这里不再赘述：

public class CommandQueue {

    private static final Map<String, Command> COMMANDS = new ConcurrentHashMap<>();

    // produce by connect、disconnect
    public static void produceCommand(String serviceName, String instanceName, Command command) {
        COMMANDS.put(serviceName + instanceName, command);
    }

    // consume by agent getCommand task
    public static Optional<Command> consumeCommand(String serviceName, String instanceName) {
        return Optional.ofNullable(COMMANDS.remove(serviceName + instanceName));
    }
}

skywalking-ui arthas console

完成了 agent 和 oap 的开发，我们再看下 ui 部分：

connect：调用oap server connect 接口，并连接 arthas-tunnel-server
disconnect：调用oap server disconnect 接口，并与 arthas-tunnel-server 断开连接
arthas 命令交互，这部分代码主要参考 arthas，大家可以查看 web-ui console 的实现

修改完skywalking-ui的代码后，我们可以直接通过 npm run dev 测试了。

如果需要通过主项目打包，别忘了在apm-webapp 的 ApplicationStartUp.java 类中添加一条 arthas 的路由：

Server
    .builder()
    .port(port, SessionProtocol.HTTP)
    .service("/arthas", oap)
    .service("/graphql", oap)
    .service("/internal/l7check", HealthCheckService.of())
    .service("/zipkin/config.json", zipkin)
    .serviceUnder("/zipkin/api", zipkin)
    .serviceUnder("/zipkin",
        FileService.of(
            ApplicationStartUp.class.getClassLoader(),
            "/zipkin-lens")
            .orElse(zipkinIndexPage))
    .serviceUnder("/",
        FileService.of(
            ApplicationStartUp.class.getClassLoader(),
            "/public")
            .orElse(indexPage))
    .build()
    .start()
    .join();

总结

BootService 启动及停止流程
如何利用 BootService 实现自定义逻辑
Agent Plugin 的类装载机制
maven-shade-plugin 的使用与注意点
如何利用 ModuleDefine 与 ModuleProvider 定义新的模块
如何向 GRPC、HTTP Service 添加新的 handler

如果你还有任何的疑问，欢迎大家与我交流。

Blog: New Features of SkyWalking Go 0.2.0

Mon, 31 Jul 2023 00:00:00 +0000

Announcing Apache SkyWalking Go 0.2.0

I’m excited to announce the release of Apache SkyWalking Go 0.2.0! This version packs several awesome new features that I’ll overview below.

Log Reporting

The log reporting feature allows the Go agent to automatically collect log content from supported logging frameworks like logrus and zap. The logs are organized and sent to the SkyWalking backend for visualization. You can see how the logs appear for each service in the SkyWalking UI:

Making Logs Searchable

You can configure certain log fields to make them searchable in SkyWalking. Set the SW_AGENT_LOG_REPORTER_LABEL_KEYS environment variable to include additional fields beyond the default log level.

For example, with logrus:

# define log with fields
logrus.WithField("module", "test-service").Info("test log")

Metrics Reporting

The agent can now collect and report custom metrics data from runtime/metrics to the backend. Supported metrics are documented here.

Automatic Instrumentation

In 0.1.0, you had to manually integrate the agent into your apps. Now, the new commands can automatically analyze and instrument projects at a specified path, no code changes needed! Try using the following command to import skywalking-go into your project:

# inject to project at current path
skywalking-go-agent -inject=./ -all

Or you can still use the original manual approach if preferred.

Get It Now!

Check out the CHANGELOG for the full list of additions and fixes. I encourage you to try out SkyWalking Go 0.2.0 today! Let me know if you have any feedback.

Zh: 使用SkyWalking go agent快速实现Dubbo Go监控

Mon, 05 Jun 2023 00:00:00 +0000

本文演示如何将 Dubbo-Go 应用程序与 SkyWalking Go 集成，并在 SkyWalking UI 中查看结果。

以前，如果你想要在 SkyWalking 中监控 Golang 应用程序，需要将项目与 go2sky 项目集成，并手动编写各种带有 go2sky 插件的框架。现在，我们有一个全新的项目（ Skywalking Go ），允许你将 Golang 项目集成到 SkyWalking 中，几乎不需要编码，同时提供更大的灵活性和可扩展性。

在本文中，我们将指导你快速将 skywalking-go 项目集成到 dubbo-go 项目中。

演示包括以下步骤：

部署 SkyWalking：这涉及设置 SkyWalking 后端和 UI 程序，使你能够看到最终效果。
使用 SkyWalking Go 编译程序：在这里，你将把 SkyWalking Go Agent 编译到要监控的 Golang 程序中。
应用部署：你将导出环境变量并部署应用程序，以促进你的服务与 SkyWalking 后端之间的通信。
在 SkyWalking UI 上可视化：最后，你将发送请求并在 SkyWalking UI 中观察效果。

部署 SkyWalking

请从官方 SkyWalking 网站下载 SkyWalking APM 程序。然后执行以下两个命令来启动服务:

# 启动 OAP 后端
> bin/oapService.sh
# 启动 UI
> bin/webappService.sh

接下来，你可以访问地址 http://localhost:8080/ 。此时，由于尚未部署任何应用程序，因此你将看不到任何数据。

使用 SkyWalking GO 编译 Dubbo Go 程序

这里将演示如何将 Dubbo-go 程序与SkyWalking Go Agent集成。请依次执行如下命令来创建一个新的项目:

# 安装dubbo-go基础环境
> export GOPROXY="https://goproxy.cn"
> go install github.com/dubbogo/dubbogo-cli@latest
> dubbogo-cli install all
# 创建demo项目
> mkdir demo && cd demo
> dubbogo-cli newDemo .
# 升级dubbo-go依赖到最新版本
> go get -u dubbo.apache.org/dubbo-go/v3

在项目的根目录中执行以下命令。此命令将下载 skywalking-go 所需的依赖项：

go get github.com/apache/skywalking-go

接下来，请分别在服务端和客户端的main包中引入。包含之后，代码将会更新为：

// go-server/cmd/server.go
package main

import (
	"context"
)

import (
	"dubbo.apache.org/dubbo-go/v3/common/logger"
	"dubbo.apache.org/dubbo-go/v3/config"
	_ "dubbo.apache.org/dubbo-go/v3/imports"

	"helloworld/api"
	
	// 引入skywalking-go
	_ "github.com/apache/skywalking-go"
)

type GreeterProvider struct {
	api.UnimplementedGreeterServer
}

func (s *GreeterProvider) SayHello(ctx context.Context, in *api.HelloRequest) (*api.User, error) {
	logger.Infof("Dubbo3 GreeterProvider get user name = %s\n", in.Name)
	return &api.User{Name: "Hello " + in.Name, Id: "12345", Age: 21}, nil
}

// export DUBBO_GO_CONFIG_PATH= PATH_TO_SAMPLES/helloworld/go-server/conf/dubbogo.yaml
func main() {
	config.SetProviderService(&GreeterProvider{})
	if err := config.Load(); err != nil {
		panic(err)
	}
	select {}
}

在客户端代码中除了需要引入skywalking-go之外，还需要在main方法中的最后一行增加主携程等待语句，以防止因为客户端快速关闭而无法将Tracing数据异步发送到SkyWalking后端：

package main

import (
	"context"
)

import (
	"dubbo.apache.org/dubbo-go/v3/common/logger"
	"dubbo.apache.org/dubbo-go/v3/config"
	_ "dubbo.apache.org/dubbo-go/v3/imports"

	"helloworld/api"

	// 引入skywalking-go
	_ "github.com/apache/skywalking-go"
)

var grpcGreeterImpl = new(api.GreeterClientImpl)

// export DUBBO_GO_CONFIG_PATH= PATH_TO_SAMPLES/helloworld/go-client/conf/dubbogo.yaml
func main() {
	config.SetConsumerService(grpcGreeterImpl)
	if err := config.Load(); err != nil {
		panic(err)
	}

	logger.Info("start to test dubbo")
	req := &api.HelloRequest{
		Name: "laurence",
	}
	reply, err := grpcGreeterImpl.SayHello(context.Background(), req)
	if err != nil {
		logger.Error(err)
	}
	logger.Infof("client response result: %v\n", reply)
	// 增加主携程等待语句
	select {}
}

接下来，请从官方 SkyWalking 网站下载 Go Agent 程序。当你使用 go build 命令进行编译时，请在 bin 目录中找到与当前操作系统匹配的代理程序，并添加 -toolexec="/path/to/go-agent -a 参数。例如，请使用以下命令：

# 进入项目主目录
> cd demo
# 分别编译服务端和客户端
# -toolexec 参数定义为go-agent的路径
# -a 参数用于强制重新编译所有依赖项
> cd go-server && go build -toolexec="/path/to/go-agent" -a -o go-server cmd/server.go && cd ..
> cd go-client && go build -toolexec="/path/to/go-agent" -a -o go-client cmd/client.go && cd ..

应用部署

在这里，我们分别启动两个终端窗口来分别启动服务端和客户端。

在服务端，将服务的名称更改为dubbo-server：

# 导出dubbo-go服务端配置文件路径
export DUBBO_GO_CONFIG_PATH=/path/to/demo/go-server/conf/dubbogo.yaml
# 导出skywalking-go的服务名称
export SW_AGENT_NAME=dubbo-server

./go-server/go-server

在客户端，将服务的名称更改为dubbo-client：

# 导出dubbo-go客户端配置文件路径
export DUBBO_GO_CONFIG_PATH=/path/to/demo/go-client/conf/dubbogo.yaml
# 导出skywalking-go的服务名称
export SW_AGENT_NAME=dubbo-client

./go-client/go-client

在 SkyWalking UI 上可视化

现在，由于客户端会自动像服务器端发送请求，现在就可以在 SkyWalking UI 中观察结果。

几秒钟后，重新访问 http://localhost:8080 的 SkyWalking UI。能够在主页上看到部署的 dubbo-server 和 dubbo-client 服务。

此外，在追踪页面上，可以看到刚刚发送的请求。

并可以在拓扑图页面中看到服务之间的关系。

总结

在本文中，我们指导你快速开发dubbo-go服务，并将其与 SkyWalking Go Agent 集成。这个过程也适用于你自己的任意 Golang 服务。最终，可以在 SkyWalking 服务中查看显示效果。如果你有兴趣了解 SkyWalking Go 代理当前支持的框架，请参阅此文档。

将来，我们将继续扩展 SkyWalking Go 的功能，添加更多插件支持。所以，请继续关注！

Blog: Quick start with SkyWalking Go Agent

Thu, 01 Jun 2023 00:00:00 +0000

Background

Previously, if you wanted to monitor a Golang application in SkyWalking, you would integrate your project with the go2sky project and manually write various frameworks with go2sky plugins. Now, we have a brand-new project (Skywalking Go) that allows you to integrate your Golang projects into SkyWalking with almost zero coding, while offering greater flexibility and scalability.

In this article, we will guide you quickly integrating the skywalking-go project into your Golang project.

Quick start

This demonstration will consist of the following steps:

Deploy SkyWalking: This involves setting up the SkyWalking backend and UI programs, enabling you to see the final effect.
Compile Golang with SkyWalking Go: Here, you’ll compile the SkyWalking Go Agent into the Golang program you wish to monitor.
Application Deployment: You’ll export environment variables and deploy the application to facilitate communication between your service and the SkyWalking backend.
Visualization on SkyWalking UI: Finally, you’ll send requests and observe the effects within the SkyWalking UI.

Deploy SkyWalking

Please download the SkyWalking APM program from the official SkyWalking website. Then execute the following two commands to start the service:

# startup the OAP backend
> bin/oapService.sh
# startup the UI
> bin/webappService.sh

Next, you can access the address at http://localhost:8080/. At this point, as no applications have been deployed yet, you will not see any data.

Compile Golang with SkyWalking GO

Here is a simple business application here that starts an HTTP service.

package main

import "net/http"

func main() {
	http.HandleFunc("/hello", func(writer http.ResponseWriter, request *http.Request) {
		writer.Write([]byte("Hello World"))
	})
	err := http.ListenAndServe(":8000", nil)
	if err != nil {
		panic(err)
	}
}

Execute the following command in the project’s root directory. This command will download the dependencies required for skywalking-go:

go get github.com/apache/skywalking-go

Also, include it in the main package of the project. After the inclusion, the code will update to:

package main

import (
	"net/http"

    // This is an important step. DON'T MISS IT.
	_ "github.com/apache/skywalking-go"
)

func main() {
	http.HandleFunc("/hello", func(writer http.ResponseWriter, request *http.Request) {
		writer.Write([]byte("Hello World"))
	})
	err := http.ListenAndServe(":8000", nil)
	if err != nil {
		panic(err)
	}
}

Next, please download the Go Agent program from the official SkyWalking website. When you compile with the go build command, find the agent program that matches your current operating system in the bin directory, and add the -toolexec="/path/to/go-agent -a parameter. For example, use the following command:

# Build application with SkyWalking go agent
# -toolexec parameter define the path of go-agent
# -a parameter is used to force rebuild all packages
> go build -toolexec="/path/to/go-agent" -a -o test .

Application Deployment

Before you start to deploy the application, you can change the service name of the current application in SkyWalking through environment variables. You can also change its configuration such as the address with the server-side. For specific details, please refer to the documentation.

Here, we’re just changing the name of the current service to demo.

# Change the service name
> export SW_AGENT_NAME=demo

Next, you can start the application:

# Start the application
> ./test

Visualization on SkyWalking UI

Now, you can send a request to the application and observe the results in the SkyWalking UI.

# Send a request
> curl http://localhost:8000/hello

After a few seconds, you can revisit the SkyWalking UI at http://localhost:8080. You will be able to see the demo service you deployed on the homepage.

Moreover, on the Trace page, you can see the request you just sent.

Conclusion

In this article, we’ve guided you to quickly develop a demo service and integrate it with SkyWalking Go Agent. This process is also applicable to your own Golang services. Ultimately, you can view the display effect in the SkyWalking service. If you’re interested in learning which frameworks the SkyWalking Go agent currently supports, please refer to this documentation.

In the future, we will continue to expand the functionality of SkyWalking Go, adding more plugin support. So, stay tuned!

Zh: SkyWalking Go Agent 快速开始指南

Thu, 01 Jun 2023 00:00:00 +0000

本文演示如何将应用程序与 SkyWalking Go 集成，并在 SkyWalking UI 中查看结果。

以前，如果你想要在 SkyWalking 中监控 Golang 应用程序，需要将项目与 go2sky 项目集成，并手动编写各种带有 go2sky 插件的框架。现在，我们有一个全新的项目（Skywalking Go ），允许你将 Golang 项目集成到 SkyWalking 中，几乎不需要编码，同时提供更大的灵活性和可扩展性。

在本文中，我们将指导你快速将 skywalking-go 项目集成到 Golang 项目中。

演示包括以下步骤：

部署 SkyWalking：这涉及设置 SkyWalking 后端和 UI 程序，使你能够看到最终效果。
使用 SkyWalking Go 编译 Golang：在这里，你将把 SkyWalking Go Agent 编译到要监控的 Golang 程序中。
应用部署：你将导出环境变量并部署应用程序，以促进你的服务与 SkyWalking 后端之间的通信。
在 SkyWalking UI 上可视化：最后，你将发送请求并在 SkyWalking UI 中观察效果。

部署 SkyWalking

请从官方 SkyWalking 网站下载 SkyWalking APM 程序。然后执行以下两个命令来启动服务:

# 启动 OAP 后端
> bin/oapService.sh
# 启动 UI
> bin/webappService.sh

接下来，你可以访问地址 http://localhost:8080/ 。此时，由于尚未部署任何应用程序，因此你将看不到任何数据。

使用 SkyWalking GO 编译 Golang

这里有一个简单的业务应用程序，启动了一个 HTTP 服务。

package main

import "net/http"

func main() {
	http.HandleFunc("/hello", func(writer http.ResponseWriter, request *http.Request) {
		writer.Write([]byte("Hello World"))
	})
	err := http.ListenAndServe(":8000", nil)
	if err != nil {
		panic(err)
	}
}

在项目的根目录中执行以下命令。此命令将下载 skywalking-go 所需的依赖项：

go get github.com/apache/skywalking-go

接下来，请将其包含在项目的 main 包中。包含之后，代码将会更新为：

package main

import (
	"net/http"
	_ "github.com/apache/skywalking-go"
)

func main() {
	http.HandleFunc("/hello", func(writer http.ResponseWriter, request *http.Request) {
		writer.Write([]byte("Hello World"))
	})
	err := http.ListenAndServe(":8000", nil)
	if err != nil {
		panic(err)
	}
}

接下来，请从官方 SkyWalking 网站下载 Go Agent 程序。当你使用 go build 命令进行编译时，请在 bin 目录中找到与当前操作系统匹配的代理程序，并添加 -toolexec="/path/to/go-agent" -a 参数。例如，请使用以下命令：

go build -toolexec="/path/to/go-agent" -a -o test .

应用部署

在这里，我们只是将当前服务的名称更改为 demo。

接下来，你可以启动应用程序：

export SW_AGENT_NAME=demo

./test

在 SkyWalking UI 上可视化

现在，向应用程序发送请求并在 SkyWalking UI 中观察结果。

几秒钟后，重新访问 http://localhost:8080 的 SkyWalking UI。能够在主页上看到部署的 demo 服务。

此外，在追踪页面上，可以看到刚刚发送的请求。

总结

在本文中，我们指导你快速开发 demo 服务，并将其与 SkyWalking Go Agent 集成。这个过程也适用于你自己的 Golang 服务。最终，可以在 SkyWalking 服务中查看显示效果。如果你有兴趣了解 SkyWalking Go 代理当前支持的框架，请参阅此文档。

将来，我们将继续扩展 SkyWalking Go 的功能，添加更多插件支持。所以，请继续关注！

Blog: SkyWalking Python Agent Supports Profiling Now

Sun, 12 Sep 2021 00:00:00 +0000

The Java Agent of Apache SkyWalking has supported profiling since v7.0.0, and it enables users to troubleshoot the root cause of performance issues, and now we bring it into Python Agent. In this blog, we will show you how to use it, and we will introduce the mechanism of profiling.

How to use profiling in Python Agent

This feature is released in Python Agent at v0.7.0. It is turned on by default, so you don’t need any extra configuration to use it. You can find the environment variables about it here.

Here are the demo codes of an intentional slow application.

import time

def method1():
    time.sleep(0.02)
    return '1'

def method2():
    time.sleep(0.02)
    return method1()

def method3():
    time.sleep(0.02)
    return method2()

if __name__ == '__main__':
    import socketserver
    from http.server import BaseHTTPRequestHandler

    class SimpleHTTPRequestHandler(BaseHTTPRequestHandler):

        def do_POST(self):
            method3()
            time.sleep(0.5)
            self.send_response(200)
            self.send_header('Content-Type', 'application/json')
            self.end_headers()
            self.wfile.write('{"song": "Despacito", "artist": "Luis Fonsi"}'.encode('ascii'))

    PORT = 19090
    Handler = SimpleHTTPRequestHandler

    with socketserver.TCPServer(("", PORT), Handler) as httpd:
        httpd.serve_forever()

We can start it with SkyWalking Python Agent CLI without changing any application code now, which is also the latest feature of v0.7.0. We just need to add sw-python run before our start command(i.e. sw-python run python3 main.py), to start the application with python agent attached. More information about sw-python can be found there.

Then, we should add a new profile task for the / endpoint from the SkyWalking UI, as shown below.

We can access it by curl -X POST http://localhost:19090/, after that, we can view the result of this profile task on the SkyWalking UI.

The mechanism of profiling

When a request lands on an application with the profile function enabled, the agent begins the profiling automatically if the request’s URI is as required by the profiling task. A new thread is spawned to fetch the thread dump periodically until the end of request.

The agent sends these thread dumps, called ThreadSnapshot, to SkyWalking OAPServer, and the OAPServer analyzes those ThreadSnapshot(s) and gets the final result. It will take a method invocation with the same stack depth and code signature as the same operation, and estimate the execution time of each method from this.

Let’s demonstrate how this analysis works through the following example. Suppose we have such a program below and we profile it at 10ms intervals.

def main():
    methodA()

def methodA():
    methodB()

def methodB():
    methodC()
    methodD()

def methodC():
    time.sleep(0.04)

def methodD():
    time.sleep(0.06)

The agent collects a total of 10 ThreadSnapShot(s) over the entire time period(Diagram A). The first 4 snapshots represent the thread dumps during the execution of function C, and the last 6 snapshots represent the thread dumps during the execution of function D. After the analysis of OAPServer, we can see the result of this profile task on the SkyWalking Rocketbot UI as shown in the right of the diagram. With this result, we can clearly see the function call relationship and the time consumption situation of this program.

Diagram A

You can read more details of profiling theory from this blog.

We hope you enjoy the profile in the Python Agent, and if so, you can give us a star on Python Agent and SkyWalking on GitHub.

Blog: Apache SkyWalking: How to propagate context between threads when using ThreadPoolExecutor

Tue, 09 Feb 2021 00:00:00 +0000

When using SkyWalking java agent, people usually propagate context easily. They even do not need to change the business code. However, it becomes harder when you want to propagate context between threads when using ThreadPoolExecutor. You can use the RunnableWrapper in the maven artifact org.apache.skywalking:apm-toolkit-trace. This way you must change your code. The developer manager usually don’t like this because there may be lots of projects, or lots of runnable code. If they don’t use SkyWalking some day, the code added will be superfluous and inelegant.

Is there a way to propagate context without changing the business code? Yes.

Skywalking java agent enhances a class by add a field and implement an interface. The ThreadPoolExecutor is a special class that is used widely. We even don’t know when and where it is loaded. Most JVMs do not allow changes in the class file format for classes that have been loaded previously. So SkyWalking should not enhance the ThreadPoolExecutor successfully by retransforming when the ThreadPoolExecutor has been loaded. However, we can apply advice to the ThreadPoolExecutor#execute method and wrap the Runnable param using our own agent, then enhance the wrapper class by SkyWalking java agent. An advice do not change the layout of a class.

Now we should decide how to do this. You can use the RunnableWrapper in the maven artifact org.apache.skywalking:apm-toolkit-trace to wrap the param, but you need to face another problem. This RunnableWrapper has a plugin whose active condition is checking if there is @TraceCrossThread. Agent core uses net.bytebuddy.pool.TypePool.Default.WithLazyResolution.LazyTypeDescription to find the annotations of a class. The LazyTypeDescription finds annotations by using a URLClassLoader with no urls if the classloader is null(bootstrap classloader). So it can not find the @TraceCrossThread class unless you change the LocationStrategy of SkyWalking java agent builder.

In this project, I write my own wrapper class, and simply add a plugin with a name match condition. Next, Let me show you how these two agents work together.

Move the plugin to the skywalking “plugins” directory.
Add this agent after the SkyWalking agent since the wrapper class should not be loaded before SkyWalking agent instrumentation have finished. For example,

java -javaagent:/path/to/skywalking-agent.jar -javaagent:/path/to/skywalking-tool-agent-v1.0.0.jar …
When our application runs
- SkyWalking java agent adds a transformer by parsing the plugin for enhancing the wrapper class in the tool agent.
- The tool agent loads the wrapper class into bootstrap classloader. This triggers the previous transformer.
- The tool agent applies an advice to the ThreadPoolExecutor class, wrapping the java.lang.Runnable param of “execute” method with the wrapper class.
- Now SkyWalking propagates the context with the wrapper class.

Enjoy tracing with ThreadPoolExecutor in SkyWalking!

Blog: Apache SkyWalking: Use Profiling to Fix the Blind Spot of Distributed Tracing

Mon, 13 Apr 2020 00:00:00 +0000

This post originally appears on The New Stack

This post introduces a way to automatically profile code in production with Apache SkyWalking. We believe the profile method helps reduce maintenance and overhead while increasing the precision in root cause analysis.

Limitations of the Distributed Tracing

In the early days, metrics and logging systems were the key solutions in monitoring platforms. With the adoption of microservice and distributed system-based architecture, distributed tracing has become more important. Distributed tracing provides relevant service context, such as system topology map and RPC parent-child relationships.

Some claim that distributed tracing is the best way to discover the cause of performance issues in a distributed system. It’s good at finding issues at the RPC abstraction, or in the scope of components instrumented with spans. However, it isn’t that perfect.

Have you been surprised to find a span duration longer than expected, but no insight into why? What should you do next? Some may think that the next step is to add more instrumentation, more spans into the trace, thinking that you would eventually find the root cause, with more data points. We’ll argue this is not a good option within a production environment. Here’s why:

There is a risk of application overhead and system overload. Ad-hoc spans measure the performance of specific scopes or methods, but picking the right place can be difficult. To identify the precise cause, you can “instrument” (add spans to) many suspicious places. The additional instrumentation costs more CPU and memory in the production environment. Next, ad-hoc instrumentation that didn’t help is often forgotten, not deleted. This creates a valueless overhead load. In the worst case, excess instrumentation can cause performance problems in the production app or overload the tracing system.
The process of ad-hoc (manual) instrumentation usually implies at least a restart. Trace instrumentation libraries, like Zipkin Brave, are integrated into many framework libraries. To instrument a method’s performance typically implies changing code, even if only an annotation. This implies a re-deploy. Even if you have the way to do auto instrumentation, like Apache SkyWalking, you still need to change the configuration and reboot the app. Otherwise, you take the risk of GC caused by hot dynamic instrumentation.
Injecting instrumentation into an uninstrumented third party library is hard and complex. It takes more time and many won’t know how to do this.
Usually, we don’t have code line numbers in the distributed tracing. Particularly when lambdas are in use, it can be difficult to identify the line of code associated with a span. Regardless of the above choices, to dive deeper requires collaboration with your Ops or SRE team, and a shared deep level of knowledge in distributed tracing.

Regardless of the above choices, to dive deeper requires collaboration with your Ops or SRE team, and a shared deep level of knowledge in distributed tracing.

Profiling in Production

Introduction

To reuse distributed tracing to achieve method scope precision requires an understanding of the above limitations and a different approach. We called it PROFILE.

Most high-level languages build and run on a thread concept. The profile approach takes continuous thread dumps. We merge the thread dumps to estimate the execution time of every method shown in the thread dumps. The key for distributed tracing is the tracing context, identifiers active (or current) for the profiled method. Using this trace context, we can weave data harvested from profiling into existing traces. This allows the system to automate otherwise ad-hoc instrumentation. Let’s dig deeper into how profiling works:

We consider a method invocation with the same stack depth and signature (method, line number etc), the same operation. We derive span timestamps from the thread dumps the same operation is in. Let’s put this visually:

Above, represents 10 successive thread dumps. If this method is in dumps 4-8, we assume it started before dump 4 and finished after dump 8. We can’t tell exactly when the method started and stopped. but the timestamps of thread dumps are close enough.

To reduce overhead caused by thread dumps, we only profile methods enclosed by a specific entry point, such as a URI or MVC Controller method. We identify these entry points through the trace context and the APM system.

The profile does thread dump analysis and gives us:

The root cause, precise to the line number in the code.
Reduced maintenance as ad-hoc instrumentation is obviated.
Reduced overload risk caused by ad-hoc instrumentation.
Dynamic activation: only when necessary and with a very clear profile target.

Implementing Precise Profiling with Apache SkyWalking 7

Distributed profiling is built-into Apache SkyWalking application performance monitoring (APM). Let’s demonstrate how the profiling approach locates the root cause of the performance issue.

final CountDownLatchcountDownLatch= new CountDownLatch(2);
 
threadPool.submit(new Task1(countDownLatch));
threadPool.submit(new Task2(countDownLatch));
 
try {
   countDownLatch.await(500, TimeUnit.MILLISECONDS);
} catch (InterruptedExceptione) {
}

Task1 and Task2 have a race condition and unstable execution time: they will impact the performance of each other and anything calling them. While this code looks suspicious, it is representative of real life. People in the OPS/SRE team are not usually aware of all code changes and who did them. They only know something in the new code is causing a problem.

To make matters interesting, the above code is not always slow: it only happens when the condition is locked. In SkyWalking APM, we have metrics of endpoint p99/p95 latency, so, we are easy to find out the p99 of this endpoint is far from the avg response time. However, this is not the same as understanding the cause of the latency. To locate the root cause, add a profile condition to this endpoint: duration greater than 500ms. This means faster executions will not add profiling load.

This is a typical profiled trace segment (part of the whole distributed trace) shown on the SkyWalking UI. We now notice the “service/processWithThreadPool” span is slow as we expected, but why? This method is the one we added the faulty code to. As the UI shows that method, we know the profiler is working. Now, let’s see what the profile analysis result say.

This is the profile analysis stack view. We see the stack element names, duration (include/exclude the children) and slowest methods have been highlighted. It shows clearly, “sun.misc.Unsafe.park” costs the most time. If we look for the caller, it is the code we added: CountDownLatch.await.

The Limitations of the Profile Method

No diagnostic tool can fit all cases, not even the profile method.

The first consideration is mistaking a repeatedly called method for a slow method. Thread dumps are periodic. If there is a loop of calling one method, the profile analysis result would say the target method is slow because it is captured every time in the dump process. There could be another reason. A method called many times can also end up captured in each thread dump. Even so, the profile did what it is designed for. It still helps the OPS/SRE team to locate the code having the issue.

The second consideration is overhead, the impact of repeated thread dumps is real and can’t be ignored. In SkyWalking, we set the profile dump period to at least 10ms. This means we can’t locate method performance issues if they complete in less than 10ms. SkyWalking has a threshold to control the maximum parallel degree as well.

Understanding the above keeps distributed tracing and APM systems useful for your OPS/SRE team.

How to Try This

Everything we discussed, including the Apache SkyWalking Java Agent, profile analysis code, and UI, could be found in our GitHub repository. We hope you enjoyed this new profile method, and love Apache SkyWalking. If so, give us a star on GitHub to encourage us.

SkyWalking 7 has just been released. You can contact the project team through the following channels:

Follow SkyWalking twitter.
Subscribe mailing list: dev@skywalking.apache.org. Send to dev-subscribe@kywalking.apache.org to subscribe to the mail list.

Co-author Sheng Wu is a Tetrate founding engineer and the founder and VP of Apache SkyWalking. He is solving the problem of observability for large-scale service meshes in hybrid and multi-cloud environments.

Adrian Cole works in the Spring Cloud team at VMware, mostly on Zipkin

Han Liu is a tech expert at Lagou. He is an Apache SkyWalking committer

Blog: Apache SkyWalking provides open source APM and distributed tracing in .NET Core field

Thu, 24 May 2018 00:00:00 +0000

Translated by Sheng Wu.

In many big systems, distributed and especially microservice architectures become more and more popular. With the increase of modules and services, one incoming request could cross dozens of service. How to pinpoint the issues of the online system, and the bottleneck of the whole distributed system? This became a very important problem, which must be resolved.

To resolve the problems in distributed system, Google published the paper “Dapper, a Large-Scale Distributed Systems Tracing Infrastructure”, which mentioned the designs and ideas of building a distributed system. Many projects are inspired by it, created in the last 10 years. At 2015, Apache SkyWalking was created by Wu Sheng as a simple distributed system at first and open source. Through almost 3 years developments, at 2018, according to its 5.0.0-alpha/beta releases, it had already became a cool open source APM system for cloud native, container based system.

At the early of this year, I was trying to build the Butterfly open source APM in .NET Core, and that is when I met the Apache SkyWalking team and its creator. I decided to join them, and cooperate with them, to provide .NET Core agent native compatible with SkyWalking. At April, I released the first version .NET core agent 0.1.0. After several weeks interation, we released 0.2.0, for increasing the stability and adding HttpClient, Database driver supports.

Before we used .NET Core agent, we need to deploy SkyWalking collector, UI and ElasticSearch 5.x. You can download the release versions at here: http://skywalking.apache.org/downloads/ and follow the docs (Deploy-backend-in-standalone-mode, Deploy-backend-in-cluster-mode) to setup the backend.

At here, I are giving a quick start to represent, how to monitor a demo distributed .NET Core applications. I can say, that is easy.

git clone https://github.com/OpenSkywalking/skywalking-netcore.git

cd skywalking-netcore

dotnet restore

dotnet run -p sample/SkyWalking.Sample.Backend dotnet run -p sample/SkyWalking.Sample.Frontend

Now you can open http://localhost:5001/api/values to access the demo application. Then you can open SkyWalking WebUI http://localhost:8080

Overview of the whole distributed system
Topology of distributed system
Application view
Trace query
Span’s tags, logs and related traces

GitHub

Website: http://skywalking.apache.org/
SkyWalking Github Repo: https://github.com/apache/incubator-skywalking
SkyWalking-NetCore Github Repo: https://github.com/OpenSkywalking/skywalking-netcore

Apache SkyWalking – Agent

Blog: SkyWalking Ruby Quick Start and Principle Introduction

Background

Deploy SkyWalking

Integration in Rails Projects

Integration in Sinatra Projects

Application Deployment

Visualization on SkyWalking UI

Plugin Implementation Mechanism

Conclusion

Zh: SkyWalking Ruby 快速开始与原理介绍

背景

部署 SkyWalking

为不同 Ruby 项目集成 SkyWalking

在 Rails 项目中集成

在 Sinatra 项目中集成

应用部署

在 SkyWalking UI 上可视化

插件实现机制

总结

Zh: CommunityOverCode Conference 2023 Asia

SkyWalking的Golang自动探针实践 刘晗

BanyanDB一个高扩展性的分布式追踪数据库 高洪涛

Zh: 将 Apache SkyWalking 与 Arthas 集成

背景介绍

connect

disconnect

agent arthas-control-plugin

oap arthas-controller-module

skywalking-ui arthas console

总结

Blog: New Features of SkyWalking Go 0.2.0

Announcing Apache SkyWalking Go 0.2.0

Log Reporting

Making Logs Searchable

Metrics Reporting

Automatic Instrumentation

Get It Now!

Zh: 使用SkyWalking go agent快速实现Dubbo Go监控

部署 SkyWalking

使用 SkyWalking GO 编译 Dubbo Go 程序

应用部署

在 SkyWalking UI 上可视化

总结

Blog: Quick start with SkyWalking Go Agent

Background

Quick start

Deploy SkyWalking

Compile Golang with SkyWalking GO

Application Deployment

Visualization on SkyWalking UI

Conclusion

Zh: SkyWalking Go Agent 快速开始指南

部署 SkyWalking

使用 SkyWalking GO 编译 Golang

应用部署

在 SkyWalking UI 上可视化

总结

Blog: SkyWalking Python Agent Supports Profiling Now

How to use profiling in Python Agent

The mechanism of profiling

Blog: Apache SkyWalking: How to propagate context between threads when using ThreadPoolExecutor

Blog: Apache SkyWalking: Use Profiling to Fix the Blind Spot of Distributed Tracing

Limitations of the Distributed Tracing

Profiling in Production

Introduction

Implementing Precise Profiling with Apache SkyWalking 7

The Limitations of the Profile Method

How to Try This

Blog: Apache SkyWalking provides open source APM and distributed tracing in .NET Core field

GitHub

SkyWalking的Golang自动探针实践刘晗

BanyanDB一个高扩展性的分布式追踪数据库高洪涛