python - Django memcache low level api giving less performance than cache middleware -

this first time posting question here please ignore deviations standard format...

here setup. nginx --> uwsgi --> django --> (memcached/ mysql)

i have set of apis in django return json response. earlier using memcached 2 part middleware auto caching. more control ,i have moved low level apis. use celery create pickled data sets , insert data in memcached once day , apis fetch data memcached.

i using nginx , uwsgi( 16 workers) in production environment , getting speed of 8k requests 2 part middleware approach. first request used slow. lower level cache getting 1/10th speed .

initially thought due middlewares , custom code getting executed, have trimmed also. moved pickle cpickle not think bottleneck. using apache benchmark conduct tests.

i attaching memcached ( -vv) logs both middleware approach , low level cache api. if can point me right direction........ thanks

i saw catche-middleware code django. seem saving django httpresponse object in cache. middleware calls low-level apis internally .i tried saving httpresponse in cache , no change...

logs

<pre>  middleware approach memcached logs <28 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >28 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >28 end <28 connection closed. <28 new auto-negotiating client connection 28: client using ascii protocol <28 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >28 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >28 end <29 new auto-negotiating client connection 29: client using ascii protocol <29 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >29 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >29 end <30 new auto-negotiating client connection 30: client using ascii protocol <30 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >30 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >30 end <32 new auto-negotiating client connection 32: client using ascii protocol <32 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >32 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >32 end <31 new auto-negotiating client connection 31: client using ascii protocol <31 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >31 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >31 end <33 new auto-negotiating client connection 33: client using ascii protocol <33 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >33 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >33 end <34 new auto-negotiating client connection 34: client using ascii protocol <34 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >34 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >34 end <35 new auto-negotiating client connection 35: client using ascii protocol <35 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >35 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >35 end <33 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >33 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >33 end <28 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >28 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >28 end <37 new auto-negotiating client connection 37: client using ascii protocol <37 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >37 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata <36 new auto-negotiating client connection 36: client using ascii protocol <36 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >36 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >36 end <40 new auto-negotiating client connection 40: client using ascii protocol <40 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >40 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >40 end <30 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >30 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >30 end <34 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >34 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >34 end <38 new auto-negotiating client connection <42 new auto-negotiating client connection 38: client using ascii protocol <38 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >38 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >38 end 42: client using ascii protocol <42 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >42 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >42 end >37 end <41 new auto-negotiating client connection <33 connection closed. <36 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata <31 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >36 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >36 end 41: client using ascii protocol <41 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >41 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >41 end >31 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >31 end <39 new auto-negotiating client connection 39: client using ascii protocol <39 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >39 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >39 end <38 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >38 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >38 end <33 new auto-negotiating client connection 33: client using ascii protocol <33 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >33 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >33 end <31 connection closed. <36 connection closed. <29 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >29 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >29 end <33 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >33 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >33 end <39 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >39 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >39 end <32 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >32 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >32 end <35 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >35 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >35 end <33 connection closed. <30 connection closed. <42 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >42 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >42 end <37 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >37 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >37 end <35 connection closed. <39 connection closed. <32 connection closed. <31 new auto-negotiating client connection 31: client using ascii protocol <31 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata <30 new auto-negotiating client connection >31 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata 30: client using ascii protocol <30 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >31 end >30 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >30 end <32 new auto-negotiating client connection 32: client using ascii protocol <32 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >32 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >32 end <32 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >32 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >32 end <30 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >30 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >30 end <31 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >31 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >31 end <33 new auto-negotiating client connection 33: client using ascii protocol <33 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >33 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >33 end <33 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >33 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >33 end <30 connection closed. <30 new auto-negotiating client connection 30: client using ascii protocol <30 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >30 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >30 end <38 connection closed. <32 connection closed. <30 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >30 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >30 end <33 connection closed. <37 connection closed. <31 connection closed. <29 connection closed. <29 new auto-negotiating client connection 29: client using ascii protocol <29 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >29 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >29 end <34 connection closed. <42 connection closed. <41 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >41 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >41 end <40 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >40 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >40 end <28 connection closed. <30 connection closed. <29 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >29 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >29 end <28 new auto-negotiating client connection 28: client using ascii protocol <28 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >28 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >28 end <28 :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >28 sending key :1:views.decorators.cache.cache_page..get.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.asia/kolkata >28 end <30 new auto-negotiating client connection <32 new auto-negotiating client connection 32: client using ascii protocol <32 :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >32 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.asia/kolkata >32 end <40 connection closed.  </pre>   ----------   ----------   low level logs <pre> <43 connection closed. <43 new auto-negotiating client connection 43: client using ascii protocol <43 :1:ls|1|0|10||| >43 sending key :1:ls|1|0|10||| >43 end <33 connection closed. <33 new auto-negotiating client connection 33: client using ascii protocol <33 :1:ls|1|0|10||| >33 sending key :1:ls|1|0|10||| >33 end <32 connection closed. <32 new auto-negotiating client connection 32: client using ascii protocol <32 :1:ls|1|0|10||| >32 sending key :1:ls|1|0|10||| >32 end <36 connection closed. <36 new auto-negotiating client connection 36: client using ascii protocol <36 :1:ls|1|0|10||| >36 sending key :1:ls|1|0|10||| >36 end <40 connection closed. <40 new auto-negotiating client connection 40: client using ascii protocol <40 :1:ls|1|0|10||| >40 sending key :1:ls|1|0|10||| >40 end <34 connection closed. <34 new auto-negotiating client connection 34: client using ascii protocol <34 :1:ls|1|0|10||| >34 sending key :1:ls|1|0|10||| >34 end <29 connection closed. <29 new auto-negotiating client connection 29: client using ascii protocol <29 :1:ls|1|0|10||| >29 sending key :1:ls|1|0|10||| >29 end <35 connection closed. <35 new auto-negotiating client connection 35: client using ascii protocol <35 :1:ls|1|0|10||| >35 sending key :1:ls|1|0|10||| >35 end <37 connection closed. <37 new auto-negotiating client connection 37: client using ascii protocol <37 :1:ls|1|0|10||| >37 sending key :1:ls|1|0|10||| >37 end <41 connection closed. <41 new auto-negotiating client connection 41: client using ascii protocol <41 :1:ls|1|0|10||| >41 sending key :1:ls|1|0|10||| >41 end <39 connection closed. <39 new auto-negotiating client connection 39: client using ascii protocol <39 :1:ls|1|0|10||| >39 sending key :1:ls|1|0|10||| >39 end <38 connection closed. <38 new auto-negotiating client connection 38: client using ascii protocol <38 :1:ls|1|0|10||| >38 sending key :1:ls|1|0|10||| >38 end <30 connection closed. <30 new auto-negotiating client connection 30: client using ascii protocol <30 :1:ls|1|0|10||| >30 sending key :1:ls|1|0|10||| >30 end <31 connection closed. <31 new auto-negotiating client connection 31: client using ascii protocol <31 :1:ls|1|0|10||| >31 sending key :1:ls|1|0|10||| >31 end <42 connection closed. <42 new auto-negotiating client connection 42: client using ascii protocol <42 :1:ls|1|0|10||| >42 sending key :1:ls|1|0|10||| >42 end <28 connection closed. <28 new auto-negotiating client connection 28: client using ascii protocol <28 :1:ls|1|0|10||| >28 sending key :1:ls|1|0|10||| >28 end <43 connection closed. <43 new auto-negotiating client connection 43: client using ascii protocol <43 :1:ls|1|0|10||| >43 sending key :1:ls|1|0|10||| >43 end <33 connection closed. <33 new auto-negotiating client connection 33: client using ascii protocol <33 :1:ls|1|0|10||| >33 sending key :1:ls|1|0|10||| >33 end <32 connection closed. <34 connection closed. <32 new auto-negotiating client connection 32: client using ascii protocol <32 :1:ls|1|0|10||| >32 sending key :1:ls|1|0|10|||  </pre> 

after trying , testing week (thanks team) ,i have realised django low level apis getting edge on database fetching not serious caching performance. django middleware class instantiated once in middleware stack throughput server process. mentioned in django official docs , django book.

on using low level apis connection initiation memcache taking time. inherited fetchfromcache middleware , created own custom middleware giving same performance 2 part middleware of django. stripping down unwanted middlewares in performance boost.