项目作者: iwanbolzern

项目描述 :
Write Cache for pymongo
高级语言: Python
项目地址: git://github.com/iwanbolzern/mongodb-write-cache.git
创建时间: 2019-06-25T09:01:11Z
项目社区:https://github.com/iwanbolzern/mongodb-write-cache

开源协议:MIT License

下载


Simple to use write Cache for pymongo

If you have many single writes to MongoDB or any other MongoDB compatible database such
as AWS DocumentDB and you have the requirement that these values must be eventually present,
this package will improve your write speed by up to 20 times. This can be used if you for example are
building a data integration service that should capture data from many individual sensors and this
data shall be stored for further analysis.

The present implementation accomplishes this by filling a write cache up to a specified threshold and then
bulk-inserts the queued values. Or waits until the flush_time is reached and inserts all values captured until
this event.

Usage

  1. from datetime import timedelta
  2. from pymongo import MongoClient
  3. from pymongo_write_cache import SyncWriteCache
  4. # create normal pymongo connection
  5. mongo_client = MongoClient('localhost', 27017)
  6. database = mongo_client['test_db']
  7. collection = database['test_col']
  8. # create a write cache with 10k buffer size and 20 seconds flush time. This means, flush my cache
  9. # after either 10k objects are received or 20 seconds have passed.
  10. cache = SyncWriteCache(collection, cache_size=10000, flush_time=timedelta(seconds=20))
  11. cache.insert_one({'my-first-document': 'yeeyy'})
  12. cache.insert_one({'my-second-document': 'yeeyy'})
  13. ...
  14. cache.flush()