One thing to do would be to use:
instead of what you've posted. See this link for more:
- Curl is an executable.
- Python is an interpreted language.
As a result, Python has a much slower "startup" time than curl, which
contributes to it's relatively slow speed despite the fact that IO is CPU
bound. This is one of the trade-offs of using an interpreted language. But,
generally while you get a relatively slow execution, the development and
maintenance time far outweighs that "loss". (Note: I said
One possible solution is as you say, to use Python to wrap curl in a
script - which is not a bad idea, but can lead to disastrous problems
(depending on usage, say deleting files), without care as there are race
conditions to consider.
Another approach is to try and decompose the original Python code into a
language like C/C++, so you can compile it and get near equivalent
performance that you desire. Examples are using shedSkin and Cython.