观看S3 Bucket和Process Files(Watch S3 Bucket and Process Files)

观看S3 Bucket和Process Files(Watch S3 Bucket and Process Files)

我如何观看S3存储桶并将上传的文件称为处理的文件夹，然后放入名为done的文件夹中？

我正在为客户端将PDF文件转换为PNG，并且希望没有处理节点在90％的时间内处于空闲状态。 如果有一种方法可以在文件上传后启动EC2实例，进行处理，然后关闭实例，那就太棒了。

任何指导都将非常感谢。

How would I watch an S3 bucket and have the files that were uploaded to say a folder called in be processed and then placed in a folder called done?

I am converting PDF files into PNGs for a client and would love to not have a processing node sitting idle 90% of the time. It would be great if there was a way to kick off an EC2 instance once a file was uploaded, do the processing, and then shut the instance down.

Any guidance in this would be greatly appreciated.

最满意答案

我的建议是这样的：

将文件上载到S3，然后立即将消息/项添加到具有需要处理的文件名的SQS队列。 根据队列的大小（等待处理的文件数）设置启动ec2实例的AWS Autoscale策略。 在启动的ec2实例处理文件后，它会从SQS队列中删除该项。 如果没有更多要处理的项目，EC2实例会自行关闭。

但是要记住一些事情，当你启动一个你需要支付整个小时的实例时，所以你不想启动实例并做几分钟的工作，然后关机或闲置 - 最好等到如果可能的话，你有足够的工作准备好处理近一个小时，即当队列增长到一定的大小时。 - （快速旋转实例，但慢慢关闭它们）

这就是高层次的想法......你认为合适。

My suggestion would be this:

Upload files to S3, and then immediately add a message/item to an SQS Queue that has the file name that needs to be processed. Setup AWS Autoscale policy that launces ec2 instances depending on the size of the Queue (the number of files waiting to be processed). After the launched ec2 instance processes a file, it deletes the item from the SQS queue. If there is no more items to be processed the EC2 instances shuts itself down.

Something to keep in mind however, when you spin up an instance you pay for the whole hour, so you don't want to spin up an instance and do a few minutes worth of work and then shutdown or sit idle - better to wait until you have enough work ready to process for close to an hour if possible i.e. when the queue grows to a certain size. - (spin instances up fast, but shut them down slowly)

Thats the high-level idea...adapt as you see fit.