Temporary File created in /tmp directory when using Stream to S3 Bucket

[akki-io]

Package Version - 3.1.0
PHP Version - 8.2

I am using the code as outlined here - https://maennchen.dev/ZipStream-PHP/guide/StreamOutput.html

We have large files (1GB+ each) in our S3 buckets that we want to zip. My understanding is that when using Stream Output
-> Stream to S3 Bucket the output archive file is directly created on remote storage i.e S3 but that is not the case. There is a file created in the tmp directory and I believe this is the output file that is getting created and once the $zipStream->finish(); is called it then uploads the output to s3.

Here is how my code looks like

$bucket = 'bucket-name';

$s3keys = [
    'sample/file1.mp4',
    'sample/file2.mp4',
    'sample/file3.mp4',
    .....
];

$client = new S3Client([.....]);
$client->registerStreamWrapper();

$zipFile = fopen("s3://$bucket/example.zip", 'w');

$zipStream = new ZipStream(
    outputStream: $zipFile,
    defaultEnableZeroHeader: true, 
    defaultCompressionMethod: CompressionMethod::STORE
);

foreach ($s3keys as $key) {
    $fileName = basename($key);
    $s3path = "s3://" . $bucket . "/" . $key;
    
    if ($streamRead = fopen($s3path, 'r')) {
       $zipStream->addFileFromStream(
           fileName: $fileName,
           stream: $streamRead,
       );

       fclose($streamRead);
    }
}

$zipStream->finish();
fclose($zipFile);

When I run this code, as stated above it create a file in the /tmp directory, so my question here is

• Is there a way not to create a file in the /tmp directory and directly create the archive on cloud storage.

So basically what I am trying to do here is, create a remote zip file from multiple remote files, without downloading the remote files on the server and also creating the zip file directly in the cloud and no tmp files on the server.

Please let me know. Or if someone can point me in the right direction, that would be much appreciated.

Thanks.

[maennchen]

I believe that this has to do with the S3 lib since ZipStream doesn’t use any tempfiles.

Can you try to stream the output somewhere else to make sure this is correct?

[maennchen]

Probably caused by this:

https://github.com/aws/aws-sdk-php/blob/596534c0627d8b38597061341e99b460437d1a16/src/S3/StreamWrapper.php#L725

Can you try opening the target file using append (a) mode instead of write (w)? I believe that should produce a different behavior of the amazon library.

[akki-io]

Thanks for the quick response. We absolutely ❤️ this package so thanks for creating it.

I tried opening the file using (a) mode instead of w unfortunately, it still produces the same result. I can still see a tmp file getting created.

Also, appreciate you sharing the AWS package link, we are investigating this further to see if we instantiate the s3 stream wrapper in a way it does not create tmp files.

[akki-io]

From the docs it says

Amazon S3 requires a Content-Length header to be specified before the payload of a request is sent. Therefore, the data to be uploaded in a PutObject operation is internally buffered using a PHP temp stream until the stream is flushed or closed.

https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/s3-stream-wrapper.html#uploading-data

So creation of those tmp files is expected when using S3-compatible cloud storage. :(

[maennchen]

@akki-io You could possibly make it work by creating a custom StreamWrapper or StreamInterface implementation that does a multipart upload to S3.

You would probably still need a small in memory buffer or tempfile to accumulate a good chunk size.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html

[maennchen]

(I would he happy for a doc PR detailing how that works if you manage to solve this.)