w3hello.com logo
Home PHP C# C++ Android Java Javascript Python IOS SQL HTML videos Categories
Python requests module GET/POST + various REST clients' requests taking longer than curl

One thing to do would be to use:

requests.get(url, stream=False)

instead of what you've posted. See this link for more:

http://docs.python-requests.org/en/latest/user/advanced/

DISCUSSION

  • Curl is an executable.
  • Python is an interpreted language.

As a result, Python has a much slower "startup" time than curl, which contributes to it's relatively slow speed despite the fact that IO is CPU bound. This is one of the trade-offs of using an interpreted language. But, generally while you get a relatively slow execution, the development and maintenance time far outweighs that "loss". (Note: I said generally).

One possible solution is as you say, to use Python to wrap curl in a script - which is not a bad idea, but can lead to disastrous problems (depending on usage, say deleting files), without care as there are race conditions to consider.

Another approach is to try and decompose the original Python code into a language like C/C++, so you can compile it and get near equivalent performance that you desire. Examples are using shedSkin and Cython.





© Copyright 2018 w3hello.com Publishing Limited. All rights reserved.