Increase max requests for Gunicorn to 5000

[mekarpeles]

At 1500 requests, If FastAPI does 50 req/sec then each worker is doing a restart every 30 seconds. @jimchamp and I were seeing particularly frequent fastapi restarts in the logs (and each restart itself takes 5-10 seconds meaning increased saturation for the remaining workers)

[Copilot]

Pull request overview

Adjusts the production Docker Compose Gunicorn configuration for the FastAPI service to reduce disruptive worker recycling under sustained load.

Changes:

• Increased Gunicorn --max-requests for the fast_web (FastAPI) service from 1500 to 5000 in production compose config.

[RayBB]

If you're actually doing 50 RPS on each worker then 5000 is still not even 2 minutes. I think this could be 10k.

It's not something Drini or I ever benchmarked it was just set to a lowish number when doing the first endpoints to be on the safe side.

The thing to keep an eye on would be memory usage. We might have memory leaks throughout the codebase. Other than that the restarts aren't particularly necessary.

[tfmorris]

@mekarpeles 10 sec downtime after 30 seconds is only 75% availability. Are you actually seeing memory leaks in practice? If not, that would suggest using a max-requests value of 0.

[mekarpeles]

Thank you @tfmorris and @RayBB -- I think as a first step I'd like to deploy the change from 1.5k to 5k with a jitter to understand how performance (and memory usage) changes along a step function. Both of your feedback is appreciated and noted and it's not inconceivable that we'd move to 0 max requests.

[mekarpeles]

Merging our compromised solution