Download performance greatly decreases when doing parallel loads

[andre-dubber]

In my project I spawn separate process for each download with HTTPotion, which works perfect on small download numbers (<5), but if I try to push higher load (~100 files) performance degrades substantially - from 20s per file to 900s per file and I start to get :timeout errors. I tried to tune performance via :ibrowse options, but didn't get much improvement.
I have put together a project that demonstrates the issue: https://github.com/andre-dubber/download_issue
Is there a way keep high throughput without distributing load over multiple Erlang nodes?

[valpackett]

I'm not sure that spawning an ibrowse worker per download is the right idea, the built-in pooling should work better. Try without the direct stuff.

(Also, you're not expecting that downloading more files in parallel would always make everything faster, right? It's only faster when the servers you're downloading from are slow.)

[andre-dubber]

Thanks for the suggestion, I have created a new branch that doesn't spawn here: https://github.com/andre-dubber/download_issue/tree/no-spawn
However issue is not going away, running 100 downloads results in only first 28 being successful and remaining are timing out.
In regards to parallel downloading it is not an attempt to speed up downloading, this is part of the service that will be serving files to many client requests, so this is how service is supposed to function.