Why does Puma delay request processing by 8–10 seconds after ALB forwards them?

Below is an explanation of what might be happening and how you can troubleshoot and monitor it.

---

1. Where are my requests getting stuck?

Even though the ALB accepts connections quickly, the delay you’re seeing (8–10 seconds) before the Rails log’s "Started" line appears suggests that the connection is waiting outside of your Rails application. Possibilities include:

• Puma’s single-threaded bottleneck: With your configuration of one worker and one thread, if one connection is handling something (or blocked), subsequent requests must wait until that thread is free.
• TCP accept queue and OS-level backlog: The request may be sitting in the OS socket’s accept queue if Puma’s process isn’t ready to accept new connections immediately.
• Puma master-level delay: The master process might be slow to dispatch accepted connections to the worker if it’s under load.

---

2. How can I find out what is causing this delay?

To isolate the cause, consider these steps:

• Review Puma’s Metrics/Stats:
  Enable Puma’s control app (if you haven’t already) to inspect real-time metrics. This can help you see if there’s a buildup of queued connections or if the worker is busy. For example, you can set up a control URL that shows the current backlog.

• Examine OS-Level Metrics:
  Check the TCP backlog settings and inspect the system’s accept queue. Commands like netstat or ss can show how many sockets are in the listening queue. For instance:

  BASHCopied
  ss -ln | grep :<your_app_port>

  Increase the backlog size if necessary via the OS’s tuning parameters.

• Monitor CPU / I/O/ Blocking:
  Use tools like strace on the Puma process to see if it’s blocking on I/O or waiting on locks. Additionally, check whether CPU/memory saturation or other resource constraints exist.

• Instrument Application Code:
  Although your logs show the actual Rails processing time is low, add logging around early request handling (e.g., at Rack middleware) to see if the delay is before Rails gets control. This can help determine if the delay is purely in the web server layer.

---

3. Is there a metric I can monitor to get alerted about such scenarios?

Yes, consider the following metrics:

• Puma Metrics:
  Instrument or expose metrics from Puma such as current connection backlog, queue length, or worker wait time (if available via the control app). You can then alert based on thresholds (e.g., if the queue length exceeds a certain size or if average wait time rises).

• ALB Metrics:
  Monitor AWS ALB metrics such as target_response_time or request_processing_time. A sudden spike in these values can indicate backlogs at the target.

• Operating System Metrics:
  Set up monitoring for the OS’s network queues using tools like CloudWatch Agent (if on EC2) to track socket queue lengths and other network stats.

• Application Performance Monitoring (APM):
  Consider using an APM tool (like New Relic, Datadog, etc.) to get detailed insights into request delays. These tools can pinpoint whether delays are occurring at the network, server or application level, and you can set up alerts accordingly.

---

By combining these monitoring strategies, you should be able to detect delays in the queue (at OS or Puma level) and correlate them with resource usage or load patterns. Also, if the single-threaded Puma worker is your bottleneck under increased load, consider modifying your Puma configuration (e.g., increasing threads) as part of your long-term solution.