Recently as part of adding a scraper to recipeyak I was investigating the various bits and pieces necessary to make the outbound network requests secure, e.g. how to avoid SSRF and other badness.
Webhooks.fyi has a nice overview of the various footguns and while proxying the outbound requests through Stripe’s Smokescreen would fix the problem, I wanted to avoid an extra service and it seemed fun to DIY it. Also it’s just a hobby project so the risk is low.
I got most things covered, but I wasn’t able to solve HTTP request timeouts.
Specifically overall timeouts, with Python’s
requests package you can set
read timeouts but you can’t say “kill this request if it takes more
than 5 seconds”.
This means when using
requests for outbound HTTP calls, an attacker could
perform a slow loris style attack where they send you a trickle of bytes over
the network which prevents the
connect timeout and
read timeout from firing
and hold up your request.
Anyways, I wanted to see how various languages solved this “total timeout” HTTP request problem.
If you’re using async Rust w/ tokio, you can use Tokio’s
With sync rust, there’s more setup involved (you have to introduce threads into the mix) but it also works.
Like Rust, Python has both sync and async io.
For async, you can use asyncio’s
With sync, I couldn’t find a non-hacky solution. If you browse stackoverflow you’ll find ideas like:
signals – which aren’t thread safe
- use gevent – that’s jumping ship to async!
- use the connect & read timeouts – they don’t do the same thing
- use multiprocessing – heavyweight, requires pickling
There’s an issue in the requests repo for this functionality, but it’s unlikely to be resolved.
I also tried fiddling with concurrent futures and their
timeout= parameter but that didn’t work - the request carried on regardless of the timeout.