How, in general, does Node.js handle 10,000 concurrent requests?

The Question :

445 people think this question is useful

I understand that Node.js uses a single-thread and an event loop to process requests only processing one at a time (which is non-blocking). But still, how does that work, lets say 10,000 concurrent requests. The event loop will process all the requests? Would not that take too long?

I can not understand (yet) how it can be faster than a multi-threaded web server. I understand that multi-threaded web server will be more expensive in resources (memory, CPU), but would not it still be faster? I am probably wrong; please explain how this single-thread is faster in lots of requests, and what it typically does (in high level) when servicing lots of requests like 10,000.

And also, will that single-thread scale well with that large amount? Please bear in mind that I am just starting to learn Node.js.

The Question Comments :
  • Because most of the work (moving data around) doesn’t involve the CPU.
  • Note also that just because there’s only one thread executing Javascript, doesn’t mean there aren’t lots of other threads doing work.
  • This question is either too broad, or a duplicate of various other questions.
  • Along with single threading, Node.js does something called as “non blocking I/O”. Here is where all the magic is done

The Answer 1

846 people think this answer is useful

If you have to ask this question then you’re probably unfamiliar with what most web applications/services do. You’re probably thinking that all software do this:

user do an action
 application start processing action
   └──> loop ...
          └──> busy processing
 end loop
   └──> send result to user

However, this is not how web applications, or indeed any application with a database as the back-end, work. Web apps do this:

user do an action
 application start processing action
   └──> make database request
          └──> do nothing until request completes
 request complete
   └──> send result to user

In this scenario, the software spend most of its running time using 0% CPU time waiting for the database to return.

Multithreaded network app:

Multithreaded network apps handle the above workload like this:

request ──> spawn thread
              └──> wait for database request
                     └──> answer request
request ──> spawn thread
              └──> wait for database request
                     └──> answer request
request ──> spawn thread
              └──> wait for database request
                     └──> answer request

So the thread spend most of their time using 0% CPU waiting for the database to return data. While doing so they have had to allocate the memory required for a thread which includes a completely separate program stack for each thread etc. Also, they would have to start a thread which while is not as expensive as starting a full process is still not exactly cheap.

Singlethreaded event loop

Since we spend most of our time using 0% CPU, why not run some code when we’re not using CPU? That way, each request will still get the same amount of CPU time as multithreaded applications but we don’t need to start a thread. So we do this:

request ──> make database request
request ──> make database request
request ──> make database request
database request complete ──> send response
database request complete ──> send response
database request complete ──> send response

In practice both approaches return data with roughly the same latency since it’s the database response time that dominates the processing.

The main advantage here is that we don’t need to spawn a new thread so we don’t need to do lots and lots of malloc which would slow us down.

Magic, invisible threading

The seemingly mysterious thing is how both the approaches above manage to run workload in “parallel”? The answer is that the database is threaded. So our single-threaded app is actually leveraging the multi-threaded behaviour of another process: the database.

Where singlethreaded approach fails

A singlethreaded app fails big if you need to do lots of CPU calculations before returning the data. Now, I don’t mean a for loop processing the database result. That’s still mostly O(n). What I mean is things like doing Fourier transform (mp3 encoding for example), ray tracing (3D rendering) etc.

Another pitfall of singlethreaded apps is that it will only utilise a single CPU core. So if you have a quad-core server (not uncommon nowdays) you’re not using the other 3 cores.

Where multithreaded approach fails

A multithreaded app fails big if you need to allocate lots of RAM per thread. First, the RAM usage itself means you can’t handle as many requests as a singlethreaded app. Worse, malloc is slow. Allocating lots and lots of objects (which is common for modern web frameworks) means we can potentially end up being slower than singlethreaded apps. This is where node.js usually win.

One use-case that end up making multithreaded worse is when you need to run another scripting language in your thread. First you usually need to malloc the entire runtime for that language, then you need to malloc the variables used by your script.

So if you’re writing network apps in C or go or java then the overhead of threading will usually not be too bad. If you’re writing a C web server to serve PHP or Ruby then it’s very easy to write a faster server in javascript or Ruby or Python.

Hybrid approach

Some web servers use a hybrid approach. Nginx and Apache2 for example implement their network processing code as a thread pool of event loops. Each thread runs an event loop simultaneously processing requests single-threaded but requests are load-balanced among multiple threads.

Some single-threaded architectures also use a hybrid approach. Instead of launching multiple threads from a single process you can launch multiple applications – for example, 4 node.js servers on a quad-core machine. Then you use a load balancer to spread the workload amongst the processes.

In effect the two approaches are technically identical mirror-images of each other.

The Answer 2

52 people think this answer is useful

What you seem to be thinking is that most of the processing is handled in the node event loop. Node actually farms off the I/O work to threads. I/O operations typically take orders of magnitude longer than CPU operations so why have the CPU wait for that? Besides, the OS can handle I/O tasks very well already. In fact, because Node does not wait around it achieves much higher CPU utilisation.

By way of analogy, think of NodeJS as a waiter taking the customer orders while the I/O chefs prepare them in the kitchen. Other systems have multiple chefs, who take a customers order, prepare the meal, clear the table and only then attend to the next customer.

The Answer 3

16 people think this answer is useful

Single Threaded Event Loop Model Processing Steps:

  • Clients Send request to Web Server.

  • Node JS Web Server internally maintains a Limited Thread pool to provide services to the Client Requests.

  • Node JS Web Server receives those requests and places them into a Queue. It is known as “Event Queue”.

  • Node JS Web Server internally has a Component, known as “Event Loop”. Why it got this name is that it uses indefinite loop to receive requests and process them.

  • Event Loop uses Single Thread only. It is main heart of Node JS Platform Processing Model.

  • Event Loop checks any Client Request is placed in Event Queue. If not then wait for incoming requests for indefinitely.

  • If yes, then pick up one Client Request from Event Queue

    1. Starts process that Client Request
    2. If that Client Request Does Not requires any Blocking IO Operations, then process everything, prepare response and send it back to client.
    3. If that Client Request requires some Blocking IO Operations like interacting with Database, File System, External Services then it will follow different approach
  • Checks Threads availability from Internal Thread Pool
  • Picks up one Thread and assign this Client Request to that thread.
  • That Thread is responsible for taking that request, process it, perform Blocking IO operations, prepare response and send it back to the Event Loop

    very nicely explained by @Rambabu Posa for more explanation go throw this Link

The Answer 4

15 people think this answer is useful

I understand that Node.js uses a single-thread and an event loop to process requests only processing one at a time (which is non-blocking).

I could be misunderstanding what you’ve said here, but “one at a time” sounds like you may not be fully understanding the event-based architecture.

In a “conventional” (non event-driven) application architecture, the process spends a lot of time sitting around waiting for something to happen. In an event-based architecture such as Node.js the process doesn’t just wait, it can get on with other work.

For example: you get a connection from a client, you accept it, you read the request headers (in the case of http), then you start to act on the request. You might read the request body, you will generally end up sending some data back to the client (this is a deliberate simplification of the procedure, just to demonstrate the point).

At each of these stages, most of the time is spent waiting for some data to arrive from the other end – the actual time spent processing in the main JS thread is usually fairly minimal.

When the state of an I/O object (such as a network connection) changes such that it needs processing (e.g. data is received on a socket, a socket becomes writable, etc) the main Node.js JS thread is woken with a list of items needing to be processed.

It finds the relevant data structure and emits some event on that structure which causes callbacks to be run, which process the incoming data, or write more data to a socket, etc. Once all of the I/O objects in need of processing have been processed, the main Node.js JS thread will wait again until it’s told that more data is available (or some other operation has completed or timed out).

The next time that it is woken, it could well be due to a different I/O object needing to be processed – for example a different network connection. Each time, the relevant callbacks are run and then it goes back to sleep waiting for something else to happen.

The important point is that the processing of different requests is interleaved, it doesn’t process one request from start to end and then move onto the next.

To my mind, the main advantage of this is that a slow request (e.g. you’re trying to send 1MB of response data to a mobile phone device over a 2G data connection, or you’re doing a really slow database query) won’t block faster ones.

In a conventional multi-threaded web server, you will typically have a thread for each request being handled, and it will process ONLY that request until it’s finished. What happens if you have a lot of slow requests? You end up with a lot of your threads hanging around processing these requests, and other requests (which might be very simple requests that could be handled very quickly) get queued behind them.

There are plenty of others event-based systems apart from Node.js, and they tend to have similar advantages and disadvantages compared with the conventional model.

I wouldn’t claim that event-based systems are faster in every situation or with every workload – they tend to work well for I/O-bound workloads, not so well for CPU-bound ones.

The Answer 5

12 people think this answer is useful

Adding to slebetman answer: When you say Node.JS can handle 10,000 concurrent requests they are essentially non-blocking requests i.e. these requests are majorly pertaining to database query.

Internally, event loop of Node.JS is handling a thread pool, where each thread handles a non-blocking request and event loop continues to listen to more request after delegating work to one of the thread of the thread pool. When one of the thread completes the work, it send a signal to the event loop that it has finished aka callback. Event loop then process this callback and send the response back.

As you are new to NodeJS, do read more about nextTick to understand how event loop works internally. Read blogs on, they were really helpful for me when I started with JavaScript/NodeJS.

The Answer 6

4 people think this answer is useful

Adding to slebetman‘s answer for more clarity on what happens while executing the code.

The internal thread pool in nodeJs just has 4 threads by default. and its not like the whole request is attached to a new thread from the thread pool the whole execution of request happens just like any normal request (without any blocking task) , just that whenever a request has any long running or a heavy operation like db call ,a file operation or a http request the task is queued to the internal thread pool which is provided by libuv. And as nodeJs provides 4 threads in internal thread pool by default every 5th or next concurrent request waits until a thread is free and once these operations are over the callback is pushed to the callback queue. and is picked up by event loop and sends back the response.

Now here comes another information that its not once single callback queue, there are many queues.

  1. NextTick queue
  2. Micro task queue
  3. Timers Queue
  4. IO callback queue (Requests, File ops, db ops)
  5. IO Poll queue
  6. Check Phase queue or SetImmediate
  7. close handlers queue

Whenever a request comes the code gets executing in this order of callbacks queued.

It is not like when there is a blocking request it is attached to a new thread. There are only 4 threads by default. So there is another queueing happening there.

Whenever in a code a blocking process like file read occurs , then calls a function which utilises thread from thread pool and then once the operation is done , the callback is passed to the respective queue and then executed in the order.

Everything gets queued based on the the type of callback and processed in the order mentioned above.


Add a Comment