Call new thread each time within large loop

问题内容:

I have 20,000 to 30,000 users, who should receive a message at a given time. SendMessage is a service that does API call against a third party site. I have this loop:

@users.each do |user|
  ...
  SendMessage.new(user.id)
  ...
end

Since there are quite large number of users, the API response takes about one second, and the last user receives the message too later than the scheduled time.

I thought of using Thread like this:

@users.each do |user|
  ...
  Thread.new{ SendMessage.new(user.id) }
  ...
end

Can I do as above? Is it a good idea to use Thread.new 20,000 times within a loop? Are there any drawbacks? Is there something else I am supposed to do?

问题评论:

    
Does it have to be synchronous?
    
no, it can be async
– Sachin srinivasan
6 hours ago

答案:

答案1:

Looking at your need to send 20,000 API calls to a third party provider, and assuming this can be taken async, you should implement this with Sidekiq or Resque.

You can issue a request initially, and then poll continuously for status update if needed.

答案评论:

    
I am using Sidekiq, but there is already similar background job running to generate reports. this has let enqueing of too many jobs and sidekiq stops. :/ . So i wanted to avoid that.
– Sachin srinivasan
6 hours ago
    
If you’re doing too much async operations and if Sidekiq doesn’t help, either scale out with Sidekiq or add RabbitMQ to your architecture.

答案2:

I can’t comment yet. But if my answer it’s not usefull I will destroy it.
So, if you use each, all records will be loaded into memory, it’s not good idea when you have more 20 000 records.
Try to use find_each. The find is performed by find_in_batches with a batch size of 1000 (or as specified by the :batch_size option).

答案评论:

    
Thank you, I will consider this.
– Sachin srinivasan
5 hours ago

原文地址:

https://stackoverflow.com/questions/47749860/call-new-thread-each-time-within-large-loop

添加评论

友情链接:蝴蝶教程