Recently as part of adding a scraper to recipeyak I was investigating the various bits and pieces necessary to make the outbound network requests secure, e.g. how to avoid SSRF and other badness.

Webhooks.fyi has a nice overview of the various footguns and while proxying the outbound requests through Stripe’s Smokescreen would fix the problem, I wanted to avoid an extra service and it seemed fun to DIY it. Also it’s just a hobby project so the risk is low.

I got most things covered, but I wasn’t able to solve HTTP request timeouts. Specifically overall timeouts, with Python’s requests package you can set connect and read timeouts but you can’t say “kill this request if it takes more than 5 seconds”.

This means when using requests for outbound HTTP calls, an attacker could perform a slow loris style attack where they send you a trickle of bytes over the network which prevents the connect timeout and read timeout from firing and hold up your request.

Anyways, I wanted to see how various languages solved this “total timeout” HTTP request problem.

Golang

Smokescreen uses Go and they use Go’s Conn.SetDeadline function to set an overall timeout.

Rust

If you’re using async Rust w/ tokio, you can use Tokio’s timeout function.

With sync rust, there’s more setup involved (you have to introduce threads into the mix) but it also works.

Python

Like Rust, Python has both sync and async io.

For async, you can use asyncio’s wait_for.

With sync, I couldn’t find a non-hacky solution. If you browse stackoverflow you’ll find ideas like:

  • use signals – which aren’t thread safe
  • use gevent – that’s jumping ship to async!
  • use the connect & read timeouts – they don’t do the same thing
  • use multiprocessing – heavyweight, requires pickling

There’s an issue in the requests repo for this functionality, but it’s unlikely to be resolved.

I also tried fiddling with concurrent futures and their timeout= parameter but that didn’t work - the request carried on regardless of the timeout.

Java

Java is sync, but unlike Python, it has a decent solution involving the standard library’s Future::get method.

JS

There are a couple options for JS, Promise.race and AbortController. Promise.race doesn’t actually cancel the request while AbortController does.

Ruby

I couldn’t find any solutions besides the timeout module, which is apparently pretty broken.

PHP

Pretty straightforward solution with PHP, use the CURLOPT_TIMEOUT option with curl_setopt.

Swift

Call .cancel on a given URLSessionTask and you’re all set in Swift land.