使用Paramiko将文件从SFTP传输到S3

本文介绍了使用Paramiko将文件从SFTP传输到S3的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我正在使用Paramiko访问远程SFTP文件夹，并且尝试编写将文件从SFTP中的路径(使用文件元数据检查其上次修改日期的简单逻辑)传输文件到AWS S3存储桶的代码.我已经使用Boto3设置了到S3的连接，但是我似乎仍然无法编写一个有效的代码来传输文件，而无需先将其下载到本地目录中.这是我使用Paramiko的 getfo()方法尝试的一些代码.但这不起作用.

I am using Paramiko to access a remote SFTP folder, and I'm trying to write code that transfers files from a path in SFTP (with a simple logic using the file metadata to check it's last modified date) to AWS S3 bucket. I have set the connection to S3 using Boto3, but I still can't seem to write a working code that transfers the files without downloading them to a local directory first. Here is some code I tried using Paramiko's getfo() method. But it doesn't work.

for f in files: # get last modified from file metadata last_modified = sftp.stat(remote_path + f).st_mtime last_modified_date = datetime.fromtimestamp(last_modified).date() if last_modified_date > date_limit: # check limit print('getting ' + f) full_path = f"{folder_path}{f}" fo = sftp.getfo(remote_path + f,f) s3_conn.put_object(Body=fo,Bucket=s3_bucket, Key=full_path)

谢谢！

推荐答案

使用 Paramiko SFTPClient.open 可以获取一个文件状对象，您可以将该对象传递给 Boto3 Client.put_object :

with sftp.open(remote_path + f, "r") as f: f.prefetch() s3_conn.put_object(Body=f)

出于 f.prefetch()的目的，请参见读取使用Python打开的文件Paramiko SFTPClient.open方法很慢 .

For the purpose of the f.prefetch(), see Reading file opened with Python Paramiko SFTPClient.open method is slow.