Recently as part of adding a scraper to recipeyak I was investigating the various bits and pieces necessary to make the outbound network requests secure, e.g. how to avoid SSRF and other badness.
Webhooks.fyi has a nice overview of the various footguns and while proxying the outbound requests through Stripe’s Smokescreen would fix the problem, I wanted to avoid an extra service and it seemed fun to DIY it. Also it’s just a hobby project so the risk is low.
I got most things covered, but I wasn’t able to solve HTTP request timeouts.
Specifically overall timeouts, with Python’s
requests package you can set
read timeouts but you can’t say “kill this request if it takes more
than 5 seconds”.
This means when using
requests for outbound HTTP calls, an attacker could
perform a slow loris style attack where they send you a trickle of bytes over
the network which prevents the
connect timeout and
read timeout from firing
and hold up your request.
Anyways, I wanted to see how various languages solved this “total timeout” HTTP request problem.
Smokescreen uses Go and they use Go’s
Conn.SetDeadline function to set an overall timeout.
If you’re using async Rust w/ tokio, you can use Tokio’s
With sync rust, there’s more setup involved (you have to introduce threads into the mix) but it also works.
Like Rust, Python has both sync and async io.
For async, you can use asyncio’s
With sync, I couldn’t find a non-hacky solution. If you browse stackoverflow you’ll find ideas like:
signals – which aren’t thread safe
- use gevent – that’s jumping ship to async!
- use the connect & read timeouts – they don’t do the same thing
- use multiprocessing – heavyweight, requires pickling
There’s an issue in the requests repo for this functionality, but it’s unlikely to be resolved.
I also tried fiddling with concurrent futures and their
timeout= parameter but that didn’t work - the request carried on regardless of the timeout.
Java is sync, but unlike Python, it has a decent solution involving the standard library’s
There are a couple options for JS,
doesn’t actually cancel the request while
I couldn’t find any solutions besides the timeout module, which is apparently pretty broken.
Pretty straightforward solution with PHP, use the
CURLOPT_TIMEOUT option with
.cancel on a given
URLSessionTask and you’re all set in Swift land.