Differences

This shows you the differences between two versions of the page.

--- bloglike:2021-06 [2021/06/17 04:50] – When AWS CodeBuild and CloudWatch logs integration doesn't work stybla
+++ bloglike:2021-06 [2021/06/25 16:02] – It's a trap and its name is FIFO queue stybla
@@ Line 39: / Line 39: @@
  --- //[[stybla@turnovfree.net|Zdenek Styblik]] 2021/06/17 09:47//
+===== It's a trap! and its name is FIFO queue =====
+When migrating from Redis-as-a-queue to AWS SQS and making application more integrated with the cloud, we've decided to use a FIFO queue. Intentions were good - ideally keep order of messages and prevent duplicates as low as possible. We were aware of the following excerpt from FAQ:
+> By design, Amazon SQS FIFO queues don't serve messages from the same message group to more than one consumer at a time. However, if your FIFO queue has multiple message groups, you can take advantage of parallel consumers
+but not so much of implications, at least for django-q. We've given it a try and it failed miserably due to combo of max attempts and buggy job running way past the limit.
+As for the implications(which I haven't verified), FIFO means that only one job can be executed at the time and nothing else. Got django-q workers/threads? Pointless. Scaling? Yes, if you emit messages into different message groups and willing to accept the fact that some messages might not be processed at all on scale down. I'm fairly sure both of these can be solved, but is it worth it? Most likely no.
+Therefore, we've switched to the simple SQS queue with no content deduplication(which probably didn't work anyway with this setup) and no FIFO and possibility of duplicates. However, that's life and it works.
+ --- //[[stybla@turnovfree.net|Zdenek Styblik]] 2021/06/25 20:56//