Active job and Sidekiq concurrency taking huge memory and performance impact

问题内容:

My code for sidekiq job. GetProcessMem gem used to get the memory usage before and after the process. It got increasing dramatically.
class TestBatchJob < ApplicationJob
queue_as :test_builder
include Math::PeriodQuarters

  def print_usage(description)
    mb = GetProcessMem.new.mb
    logger.info "#{ description } - MEMORY USAGE(MB): #{ mb.round }"
  end

  def perform
    res = Benchmark.bm do |x|
      x.report("Perform task timeing") do
        print_usage("Before")
        names = []
        10_000_000.times do |item|
          names << "Name #{item}"
        end
        print_usage("After")
      end
    end
    logger.info "Benchmark result #{res.first.total.to_s}"
  end
end

When running single worker

[b890e994-582a-44ab-af45-22e59aee5455] Performing TestBatchJob (Job ID: b890e994-582a-44ab-af45-22e59aee5455) from Sidekiq(test_builder)
[b890e994-582a-44ab-af45-22e59aee5455] Before - MEMORY USAGE(MB): 1220
[b890e994-582a-44ab-af45-22e59aee5455] After - MEMORY USAGE(MB): 1220
[b890e994-582a-44ab-af45-22e59aee5455] Benchmark result 4.529999999999996
[b890e994-582a-44ab-af45-22e59aee5455] Performed TestBatchJob (Job ID: b890e994-582a-44ab-af45-22e59aee5455) from Sidekiq(test_builder) in 4546.21ms

Running 10 concurrency in sidekiq

[342e9a9f-dc4f-4eec-9fce-92f1f33517db] Before - MEMORY USAGE(MB): 724
[10b11254-6900-4dbd-b1e0-ddab3d384174] Before - MEMORY USAGE(MB): 724
[e6bbcc06-01bf-4d57-8d70-46fd076ec07f] Before - MEMORY USAGE(MB): 724
[09a03dee-28c7-4722-9637-5c70b319c584] Before - MEMORY USAGE(MB): 724
[31a2c2e3-47ca-48f5-95bb-45a55a54bc8f] Before - MEMORY USAGE(MB): 724
[5757965b-adab-4d91-ac90-ed66224c6bcc] Before - MEMORY USAGE(MB): 724
[018fbd1f-2379-4ebf-a7ca-f6f9335d9002] Before - MEMORY USAGE(MB): 724
[81121b45-1cf5-41ff-a0cb-d70eca6546e0] Before - MEMORY USAGE(MB): 724
[dc1c2ae5-abc9-444a-8ed8-ff2036930990] Before - MEMORY USAGE(MB): 732
[10b11254-6900-4dbd-b1e0-ddab3d384174] After - MEMORY USAGE(MB): 4379
[342e9a9f-dc4f-4eec-9fce-92f1f33517db] After - MEMORY USAGE(MB): 4431
[e6bbcc06-01bf-4d57-8d70-46fd076ec07f] After - MEMORY USAGE(MB): 4607
[342e9a9f-dc4f-4eec-9fce-92f1f33517db] Benchmark result 79.66999999999999
[10b11254-6900-4dbd-b1e0-ddab3d384174] Benchmark result 78.84000000000002
[e6bbcc06-01bf-4d57-8d70-46fd076ec07f] Benchmark result 80.36999999999999
[09a03dee-28c7-4722-9637-5c70b319c584] After - MEMORY USAGE(MB): 4763
[342e9a9f-dc4f-4eec-9fce-92f1f33517db] Performed TestBatchJob (Job ID: 342e9a9f-dc4f-4eec-9fce-92f1f33517db) from Sidekiq(test_builder) in 127633.75ms
[31a2c2e3-47ca-48f5-95bb-45a55a54bc8f] After - MEMORY USAGE(MB): 4820
[10b11254-6900-4dbd-b1e0-ddab3d384174] Performed TestBatchJob (Job ID: 10b11254-6900-4dbd-b1e0-ddab3d384174) from Sidekiq(test_builder) in 132140.78ms
[5757965b-adab-4d91-ac90-ed66224c6bcc] After - MEMORY USAGE(MB): 4902
[018fbd1f-2379-4ebf-a7ca-f6f9335d9002] After - MEMORY USAGE(MB): 4896
[e6bbcc06-01bf-4d57-8d70-46fd076ec07f] Performed TestBatchJob (Job ID: e6bbcc06-01bf-4d57-8d70-46fd076ec07f) from Sidekiq(test_builder) in 132739.1ms
[81121b45-1cf5-41ff-a0cb-d70eca6546e0] After - MEMORY USAGE(MB): 4910
[09a03dee-28c7-4722-9637-5c70b319c584] Benchmark result 82.94
[31a2c2e3-47ca-48f5-95bb-45a55a54bc8f] Benchmark result 83.18
[5757965b-adab-4d91-ac90-ed66224c6bcc] Benchmark result 84.99
[018fbd1f-2379-4ebf-a7ca-f6f9335d9002] Benchmark result 85.02999999999999
[09a03dee-28c7-4722-9637-5c70b319c584] Performed TestBatchJob (Job ID: 09a03dee-28c7-4722-9637-5c70b319c584) from Sidekiq(test_builder) in 137796.23ms
[dc1c2ae5-abc9-444a-8ed8-ff2036930990] After - MEMORY USAGE(MB): 4993
[81121b45-1cf5-41ff-a0cb-d70eca6546e0] Benchmark result 85.08
[31a2c2e3-47ca-48f5-95bb-45a55a54bc8f] Performed TestBatchJob (Job ID: 31a2c2e3-47ca-48f5-95bb-45a55a54bc8f) from Sidekiq(test_builder) in 137797.17ms
[5757965b-adab-4d91-ac90-ed66224c6bcc] Performed TestBatchJob (Job ID: 5757965b-adab-4d91-ac90-ed66224c6bcc) from Sidekiq(test_builder) in 137801.78ms
[018fbd1f-2379-4ebf-a7ca-f6f9335d9002] Performed TestBatchJob (Job ID: 018fbd1f-2379-4ebf-a7ca-f6f9335d9002) from Sidekiq(test_builder) in 137796.29ms
[dc1c2ae5-abc9-444a-8ed8-ff2036930990] Benchmark result 85.00999999999999
[81121b45-1cf5-41ff-a0cb-d70eca6546e0] Performed TestBatchJob (Job ID: 81121b45-1cf5-41ff-a0cb-d70eca6546e0) from Sidekiq(test_builder) in 137796.56ms
[dc1c2ae5-abc9-444a-8ed8-ff2036930990] Performed TestBatchJob (Job ID: dc1c2ae5-abc9-444a-8ed8-ff2036930990) from Sidekiq(test_builder) in 137797.2ms

I am running this with redis-server. How can I debug this issue.

问题评论:

原文地址:

https://stackoverflow.com/questions/47753939/active-job-and-sidekiq-concurrency-taking-huge-memory-and-performance-impact

添加评论