HTTP Timeouts
Recently as part of adding a scraper to recipeyak I was investigating the various bits and pieces necessary to make the outbound network requests secure, e.g. how to avoid SSRF and other badness.
Webhooks.fyi has a nice overview of the various footguns and while proxying the outbound requests through Stripe’s Smokescreen would fix the problem, I wanted to avoid an extra service and it seemed fun to DIY it. Also it’s just a hobby project so the risk is low.
I got most things covered, but I wasn’t able to solve HTTP request timeouts.
Specifically overall timeouts, with Python’s requests
package you can set
connect
and read
timeouts but you can’t say “kill this request if it takes more
than 5 seconds”.
This means when using requests
for outbound HTTP calls, an attacker could
perform a slow loris style attack where they send you a trickle of bytes over
the network which prevents the connect
timeout and read
timeout from firing
and hold up your request.
Anyways, I wanted to see how various languages solved this “total timeout” HTTP request problem.
Golang
Smokescreen uses Go and they use Go’s Conn.SetDeadline
function to set an overall timeout.
Rust
If you’re using async Rust w/ tokio, you can use Tokio’s timeout
function.
With sync rust, there’s more setup involved (you have to introduce threads into the mix) but it also works.
Python
Like Rust, Python has both sync and async io.
For async, you can use asyncio’s wait_for
.
With sync, I couldn’t find a non-hacky solution. If you browse stackoverflow you’ll find ideas like:
- use
signal
s – which aren’t thread safe - use gevent – that’s jumping ship to async!
- use the connect & read timeouts – they don’t do the same thing
- use multiprocessing – heavyweight, requires pickling
There’s an issue in the requests repo for this functionality, but it’s unlikely to be resolved.
I also tried fiddling with concurrent futures and their timeout=
parameter but that didn’t work - the request carried on regardless of the timeout.
Java
Java is sync, but unlike Python, it has a decent solution involving the standard library’s Future::get
method.
JS
There are a couple options for JS, Promise.race
and AbortController
. Promise.race
doesn’t actually cancel the request while AbortController
does.
Ruby
I couldn’t find any solutions besides the timeout module, which is apparently pretty broken.
PHP
Pretty straightforward solution with PHP, use the CURLOPT_TIMEOUT
option with curl_setopt
.
Swift
Call .cancel
on a given URLSessionTask
and you’re all set in Swift land.