Celery

Recently we faced an issue how to fairly distribute computational resources for our imaging service using Celery. The imaging service does some basic operations with image like resize. We chose Celery to distribute work among our workers. The service works with chunks of 1000 images that is processed in batch. We wanted to avoid situation where one large client uses all processors on a worker and other smaller clients would have to wait. To achieve this we needed to create special queue for each client. Celery worker then takes tasks from all queues in round robin fashion. The problem is that we do not know what clients will be using our service in future and thus we needed to add queues for them dynamically. To dynamically add new queue and more consumer for it call following method: ...