Celery

Recently we faced an issue how to fairly distribute computational resources for our imaging service using Celery. The imaging service does some basic operations with image like resize. We chose Celery to distribute work among our workers. The service works with chunks of 1000 images that is processed in batch. We wanted to avoid situation where one large client uses all processors on a worker and other smaller clients would have to wait....