I desperately need some devops/scaling help. Why is AWS lambda *slower* with 100 concurrent requests than my express server??

you desperately need a new sack of potatoes bc the one you're wearing is all worn out on the lambda front, I bet the startup is what's killing you, perhaps it'd be better to have the express/fastify/bun/elysia server as your main entrypoint and loadbalancing to lambda if necessary?

I love this hoodie. Hmmmm, but isn't aws supposed to scale well? And it only is slow like this with 100 concurrent calls. 50 seems fine

I’d argue it scales well in the sense that it’ll spin up as many lambdas as needed, but each spin up comes with a cold start when it’s not constant traffic you’re getting but spikes, so your bottleneck is startup delay do you have observability set up?

Hmmm, I don't think the cold startup is *that* bad though. Like what I'm seeing I don't think is a cold start problem. I don't know what observability is

cloudflare posted this video about their load balancer today looks easy enough to set up haven't looked at pricing but without looking I'm sure it's at least 1/50 the price of aws'

Thanks!

if I may ask, did you find out what the cause was?

No im still baffled. Gonna start talking to some devops people this weekend

What is the code doing? Are you getting 500s because it’s scraping something and they are blocking known AWS ip ranges? What if you change the lambdas to hit a different server, maybe one you control. Could also be cold starts or not enough mem / CPU allocated? Or maybe account level limits on concurrency that need bumped?

@stetsblake Well the 502s are coming from my exisiting setup on render + express.

I know he's super busy but @twigpress has forgotten more about Lambda than most of us will ever learn.

@twigpress Thank you for the rec! @twigpress what am I doing wrong?

First try fixing where you are now, some. So make the current situation better, buying time while you possibly move (but depends on data). The goal is to get rid of some of the 502s immediately, working on a longer term complete solution long term. This means I'd stick with AWS little longer (you're already there!) before moving onto something else like GCP. 1. try increasing memory so the cold starts are as bad, start from 2G and move it up. CPU will/should scale with it (men). 2. next setup a baseline of warm functions with provisioned concurrency sets, this won't solve all your issues but WILL get rid of some of those 502s. By this point hopefully you got rid of some 5xx responses. From here I would think about ECS, which can predictably scale AND boots up much quicker. See docs here: Me <-- Professional DevOps Engineer with Practical Business Sense.

Oh I'm using render now and want to switch to AWS lambda. Thanks so much for your response. So increase memory and how do I do number 2?

So my recommendations are for aws. (I would move off render once you have validated market fit, and you want to take over the reigns some with infra etc). #2 docs:

Cold starts for serverless is a problem.

Hmmm, chat gpt is telling me it doesn't think its a cold start problem :/

Are you running 1 lambda and 100 concurrency or 100 lambda requests?

I'm not sure honestly, I'm so bad at this stuff. I have 1 lambda function and I'm hitting it 100 times at once. And all of them finish slower than my express server, which I am shocked by. 50 concurrent calls works great on the lambda function

There’s a lot going on with lambdas and many ways to optimize. Need to find you an aws devops freelancer to help.

I use lambda extensively. you don't need to request a concurrency increase for the amount of volume you're running in your test. you should expect a burst like this to run slower than if it was handled by one beefy server, that is the trade-off. your code is the issue.

Hmmmm, but if it runs fine on the express server, why wouldn’t it run fine on lambda? I just don’t know what to change in the code

writing code for a framework is not quite the same as writing for a lambda. for the latter you need to think of it as running with minimal resources and then shutting down at the end cleanly.

also, aws has a pretty good log system, so just console.log() that shizz until you find the issue.

Because Lambda is a shim runtime not meant for production usecases We built a real, elastic, runtime for this I am biased, but, Railway is the solution you are 100% looking for. -&gt; point to Git repo -&gt; done

ajuda o cabeça de batata @acgfbr

cold starts slow down your first requests, that's probably part of it. You can "ping" your lambda every minute or so to keep it active (the cost is probably not that much). Depending on the size of your lambda (memory) you are also sharing compute with other lambdas, so you might have to get more memory so that it runs unimpeded (I think the min is 1gb) There is a concurrency limit that you can set on lambdas. setting up your own vps is the cheapest. has pretty good pricing. I don't remember how much memory/compute express js uses, but I do know hono is way more efficient so you might not need as big an instance (probably a moot point since you're not going to change your backend now).

oh boy

I need help Shayan

I honestly have no idea what's going on. Are these sent as a single request? what do they do? Also what is that thing you're wearing?

Had it since 2009. Yeah 100 requests to my scrape tiktok profile endpoint. The express server is way faster than my lambda function

Try Railway?

Thats just basically the same as render isn't it?

Possibly. They have all their own infra. They own the metal. I have been loving it but can't say for sure it's the best for this. @thisismahmoud or @JustJake might have thoughts.

👀

Any thoughts?

yeah, so: your express box is 1 hot process doing 100 requests. Lambda is 100 cold-ish containers, so it was always going to be slower fix: 1. don’t run 100 scrapes synchronously, API → SQS → Lambda 2. let Lambda drain SQS at 50–100 concurrency 3. give the function 1–2GB RAM so init is faster 4. if you must stay synchronous, provision concurrency = 100 basically: queue it or pre-warm it. you just hit cold starts + concurrency ramp

Mmmmm, but I continue to call it at 100 over and over again and it still is slow. Wouldn't it be warm after the first couple times I run it?

I would recommend getting support from Render first so they can help you figure out whats going on

Not a bad idea

Actually i just realized if running a scraping service is against their ToS, i would think twice before reaching out. Yeah probably get an indie devops guy to help out.

They've helped me many times before

Okay perfect, def reach out then

Reach out to Jesse Oullete he has literally become an API master Lead magic is insane.

How do I get in touch with him?

LinkedIn DM he's a class act

This guy?

Yes sir!

kinda off topic, but i gave up on aws/gcp, i tried a GCP instance + cloud file store and was like 20x the price for my own vps plus the confusing pricing and not much faster. i think private cloud is the future until you need real service level agreements

Interesting. I am scared of all the configuration I'd have to do in my own VPS. I am terrible at dev ops and server stuff

@fashionluvr Have a look into kamal, I’m running it in production it’s flawless. All in all a server’s just a computer, nothing to be scared of 😎 what scares you the most about it?

@fashionluvr I don't know all the lingo and configuration I need to keep a server up

@PaulHenri_L I think chatgpt/gemini you can ask any question and tells you what to do. for a background job, vps is perfect

Hey Adrian, I just sent a dm, I think I might be able to help you out

Have you tried Vercel? ;)

lol

Did you get into a fight? What happened to your sweater?

I love this hoodie. Had it since 2009

Check your account's Lambda concurrency limit in the Lambda dashboard. The default limit is relatively low to prevent abuse and you might need to request an increase if that is the issue.

This thing? I think its already set to 1k

Yeah that looks fine. Would need more details on what your code is doing to diagnose more. For example, if you're connecting to a database your single Express server would pool connections more efficiently vs Lambda making 100 separate connections.

Yeah that might be part of it

Have you though about using azure functions? I have had way better results in terms of concurrent requests with minimal cold restarts. You can keep instances pre-warmed so you don’t get hit with those annoying delays. Plus each instance can handle multiple requests at once, whereas Lambda spins up a separate container for each concurrent execution. Worth a shot if you’re dealing with performance issues. Plus I’m pretty sure it’s going to be cheaper for your use case

Nah I'm not very familiar with Azure. And its microsoft 🥴 gross lol

🤣

request an increase to the concurrency limit, can ask for 1000. if you have long running processes it can use it up quickly.

In lambda? I increase the concurrency limit? This thing? Already says 1000

Hmmm, but I'm at 1000, and I make 100 concurrent calls, no where close to 1000 right?

just to double check... Im referencing the "service quota" page. you can search for it in the search box at the top of the aws console. Im pretty sure new accounts start with 100 concurrent executions.

Looks like mine says 1000. But yeah nice to check

How much did you pay for this setup in aws?

0 😅 I'm doing it myself

My two cents take here is that anything that will hold/take a lot resources is not made for serverless

Oh you don't think scrapers are good in serverless? What do you use?

Oh fuck me.

Let us know if you have any wins? Always interested to learn more about this stuff 💪

because you are not running 100 AWS lambdas concurrently. You send 1 event to 1 Lambda. Just invoke it like: for i in {1..100}; do aws lambda invoke \ --function-name myFunction \ --invocation-type Event \ --payload "{\"id\": $i}" \ response$i.json &amp; done wait

Try railway's bun function?

@ChrisMorancie

Don’t try to scale with lambdas, you waste a lot of time and resources with boot times and need to pay a huge premium for the resources (cpu/memory). If you want to scale this for 1/10th the cost, dm me. I live in Austin too, this is my jam

Hey Adrian, I’d be happy to take a look at your setup and offer some advice free of charge. I’ve worked with Lambda for several years and there are some configuration pieces that you should consider. Several other replies here that are helpful, I recommend using AWS X-ray also

What are your lambda concurrency limits on your project? And if it’s greater than 100 are these lambdas possibly calling other resources that are pushing you over limits somewhere?