indexing_merge #

描述 #

indexing_merge 处理器用来消费队列里面的纯 JSON 文档，并合并成 Bulk 请求保存到指定的队列里面，需要配合 bulk_indexing 处理器进行消费，用批量写入代替单次请求来提高写入吞吐。

配置示例 #

一个简单的示例如下：

pipeline:
  - name: indexing_merge
    auto_start: true
    keep_running: true
    processor:
      - indexing_merge:
          input_queue: "request_logging"
          elasticsearch: "logging-server"
          index_name: "infini_gateway_requests"
          output_queue:
            name: "gateway_requests"
            label:
              tag: "request_logging"
          worker_size: 1
          bulk_size_in_mb: 10
  - name: logging_requests
    auto_start: true
    keep_running: true
    processor:
      - bulk_indexing:
          bulk:
            compress: true
            batch_size_in_mb: 10
            batch_size_in_docs: 5000
          consumer:
            fetch_max_messages: 100
          queues:
            type: indexing_merge
          when:
            cluster_available: [ "logging-server" ]

参数说明 #