User Tools

Site Tools


bloglike:2021-06

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
bloglike:2021-06 [2021/06/25 16:02] – It's a trap and its name is FIFO queue styblabloglike:2021-06 [2021/06/29 03:51] (current) – How to run django-q or similar at AWS Elastic Beanstalk stybla
Line 49: Line 49:
 but not so much of implications, at least for django-q. We've given it a try and it failed miserably due to combo of max attempts and buggy job running way past the limit. but not so much of implications, at least for django-q. We've given it a try and it failed miserably due to combo of max attempts and buggy job running way past the limit.
  
-As for the implications(which I haven't verified), FIFO means that only one job can be executed at the time and nothing else. Got django-q workers/threads? Pointless. Scaling? Yes, if you emit messages into different message groups and willing to accept the fact that some messages might not be processed at all on scale down. I'm fairly sure both of these can be solved, but is it worth it? Most likely no.+As for the implications(which I haven't verified), FIFO means that only one job can be executed at the time and nothing else. Do you have more than just one worker/thread in django-q? Pointless. Scaling based on "queue length"? Yes, if you emit messages into different message groups and willing to accept the fact that some messages might not be processed at all on scale down. I'm fairly sure both of these can be solved, but is it worth it? Most likely no.
  
 Therefore, we've switched to the simple SQS queue with no content deduplication(which probably didn't work anyway with this setup) and no FIFO and possibility of duplicates. However, that's life and it works. Therefore, we've switched to the simple SQS queue with no content deduplication(which probably didn't work anyway with this setup) and no FIFO and possibility of duplicates. However, that's life and it works.
  
  --- //[[stybla@turnovfree.net|Zdenek Styblik]] 2021/06/25 20:56//  --- //[[stybla@turnovfree.net|Zdenek Styblik]] 2021/06/25 20:56//
 +
 +
 +===== How to run django-q or similar at AWS Elastic Beanstalk =====
 +
 +This is like one year late and doesn't bring anything new nowadays(I'll explain why in a bit). I've seen couple questions here and there about how to run [[https://django-q.readthedocs.io/en/latest/index.html|django-q]] at AWS Elastic Beanstalk(EBT) and as a matter of fact, I was facing the same conundrum.
 +
 +EBT offers "web application" and "worker" environments and django-q falls, or fits into, the latter rather than former, but:
 +
 +> When you launch a worker environment, Elastic Beanstalk installs the necessary support files for your programming language of choice and a daemon on each EC2 instance in the Auto Scaling group. **The daemon reads messages from an Amazon SQS queue. The daemon sends data from each message that it reads to the web application running in the worker environment for processing.**
 +
 +Daemon passing messages from the queue as HTTP requests? That won't fly with django-q(at least I'm not aware such thing is possible) and it doesn't fit into web application -> queue -> worker(s)/django-q architecture. Also, my customer used Redis as a queue in "runs at laptop" PoC(Redis has been replaced by SQS later on).
 +
 +Ok, "worker" environment is a no-go. However, if you deploy django-q into "web application" environment, and there is nothing stopping you to do so, health of EBT application(environment) will be permanently in red. Solution? Simple wrapper with HTTP server and health endpoint. You can find such example in django-q's [[https://django-q.readthedocs.io/en/latest/examples.html#http-health-check|documentation]] now. No, unfortunately I haven't contributed that. No, that's not where I got this idea and what not. Why? Because it wasn't there one year ago ;)
 +
 + --- //[[stybla@turnovfree.net|Zdenek Styblik]] 2021/06/29 08:23//
bloglike/2021-06.txt · Last modified: 2021/06/29 03:51 by stybla