Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inconsistent uploads to S3 folders and subfolder structures despite 204 #13

Open
brian-carnot opened this issue May 3, 2024 · 4 comments
Open

Inconsistent uploads to S3 folders and subfolder structures despite 204 #13

brian-carnot opened this issue May 3, 2024 · 4 comments

Comments

@brian-carnot
Copy link

I managed to recreate the behavior with the following, but as more and more folders get added it becomes hard for me to fully track the pattern.

I have 1 bucket:

s3://user-files/

The first time I upload something I do it to the following pattern:

s3://user-files/UUID_1/UUID_2/file_name

This returns 204 but the file never shows up. If I try to delete it, I get a 404.

However, if I upload something to either s3://user-files/UUID_1/ or s3://user-files/UUID_1/UUID_2/ then it works the following time and all following times.

If I pre-create s3://user-files/UUID_1/UUID_2/ then the file upload never fails.
@GREsau
Copy link
Owner

GREsau commented May 3, 2024

I was unable to reproduce this, but it's not clear to me how you're actually uploading these files (I was using the S3 PutObject API).

How are you uploading these files?
What version of localstack-persist are you using?
Are there any errors in the localstack-persist logs? (if not, try setting DEBUG=1 as an environment variable on the container and see if anything interesting is logged then)

@brian-carnot
Copy link
Author

  1. I'm using a lambda function to get a presigned URL to POST to from my react frontend:
      const response = await axios.post(s3Url, {
        userId: userId,
        rawPath: `${workflowId}/${moduleId}/${file.name}`,
      });
      
      const { url, fields } = response.data;

      console.log('Got presigned URL:', url, fields)

      // Form Data to send to S3
      const formData = new FormData();
      Object.entries(fields).forEach(([key, value]) => {
        formData.append(key, value as string);
      });
      formData.append('file', file);

      // Upload the file to S3
      const uploadResponse = await axios.post(url, formData, {
        headers: {
          'Content-Type': 'multipart/form-data',
        },
      });
  1. I'm using localstack-persist v3.4.0
  2. I don't notice any errors. The following is my FIRST call from a fresh localstack state. It returns AWS s3.PostObject => 204, but the object won't be actually uploaded to the s3 bucket:
2024-05-07 15:22:41 2024-05-07T06:22:41.417 DEBUG --- [ead-4 (_run)] localstack_persist.state   : Nothing to persist - no services were changed
2024-05-07 15:22:45 2024-05-07T06:22:45.888 DEBUG --- [   asgi_gw_2] rolo.gateway.wsgi          : OPTIONS xuimwon6h1hmgh5nzi3j959v7rqq4ymd.lambda-url.us-east-1.localhost.localstack.cloud:4566/
2024-05-07 15:22:45 2024-05-07T06:22:45.898  INFO --- [   asgi_gw_2] localstack.request.http    : OPTIONS / => 204
2024-05-07 15:22:45 2024-05-07T06:22:45.903 DEBUG --- [   asgi_gw_3] rolo.gateway.wsgi          : POST xuimwon6h1hmgh5nzi3j959v7rqq4ymd.lambda-url.us-east-1.localhost.localstack.cloud:4566/
2024-05-07 15:22:45 2024-05-07T06:22:45.908 DEBUG --- [   asgi_gw_3] l.s.l.i.version_manager    : Got an invocation for function arn:aws:lambda:us-east-1:000000000000:function:presign:$LATEST with request_id 44e27d76-a832-4f00-9070-519ab21e3cf8
2024-05-07 15:22:45 2024-05-07T06:22:45.910 DEBUG --- [   asgi_gw_3] l.s.l.i.assignment         : Starting new environment
2024-05-07 15:22:45 2024-05-07T06:22:45.911 DEBUG --- [   asgi_gw_3] l.s.l.i.docker_runtime_exe : Assigning container name of jinbaflow-localstack-main-lambda-presign-ddacb0e247c671706a6212e98d410f01 to executor ddacb0e247c671706a6212e98d410f01
2024-05-07 15:22:45 2024-05-07T06:22:45.930 DEBUG --- [   asgi_gw_0] rolo.gateway.wsgi          : POST localhost:4566/
2024-05-07 15:22:46 2024-05-07T06:22:46.086 DEBUG --- [   asgi_gw_3] l.u.c.docker_sdk_client    : Copying file /usr/lib/localstack/lambda-runtime/v0.1.28-pre/arm64/. into jinbaflow-localstack-main-lambda-presign-ddacb0e247c671706a6212e98d410f01:/
2024-05-07 15:22:46 2024-05-07T06:22:46.700 DEBUG --- [   asgi_gw_3] l.u.c.docker_sdk_client    : Copying file /tmp/lambda/awslambda-us-east-1-tasks/presign-002d825c-3d95-4e3a-b458-5f9e15c8eff7/code/. into jinbaflow-localstack-main-lambda-presign-ddacb0e247c671706a6212e98d410f01:/var/task
2024-05-07 15:22:46 2024-05-07T06:22:46.770 DEBUG --- [   asgi_gw_3] l.u.c.docker_sdk_client    : Starting container jinbaflow-localstack-main-lambda-presign-ddacb0e247c671706a6212e98d410f01
2024-05-07 15:22:48 2024-05-07T06:22:48.715 DEBUG --- [   asgi_gw_1] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-05-07 15:22:48 2024-05-07T06:22:48.734 DEBUG --- [   asgi_gw_3] l.u.c.container_client     : Getting ipv4 address for container jinbaflow-localstack-main-lambda-presign-ddacb0e247c671706a6212e98d410f01 in network backend_default.
2024-05-07 15:22:50 2024-05-07T06:22:50.337 DEBUG --- [   asgi_gw_0] rolo.gateway.wsgi          : POST 172.19.0.6:4566/_localstack_lambda/ddacb0e247c671706a6212e98d410f01/status/ddacb0e247c671706a6212e98d410f01/ready
2024-05-07 15:22:50 2024-05-07T06:22:50.340 DEBUG --- [   asgi_gw_3] l.s.l.i.execution_environm : Start of execution environment ddacb0e247c671706a6212e98d410f01 for function arn:aws:lambda:us-east-1:000000000000:function:presign:$LATEST took 4429.13ms
2024-05-07 15:22:50 2024-05-07T06:22:50.342  INFO --- [   asgi_gw_0] localstack.request.http    : POST /_localstack_lambda/ddacb0e247c671706a6212e98d410f01/status/ddacb0e247c671706a6212e98d410f01/ready => 202
2024-05-07 15:22:50 2024-05-07T06:22:50.343 DEBUG --- [   asgi_gw_3] l.s.l.i.docker_runtime_exe : Sending invoke-payload '{"invoke-id": "44e27d76-a832-4f00-9070-519ab21e3cf8", "invoked-function-arn": "arn:aws:lambda:us-east-1:000000000000:function:presign", "payload": "{\"version\": \"2.0\", \"routeKey\": \"$default\", \"rawPath\": \"/\", \"rawQueryString\": \"\", \"headers\": {\"host\": \"xuimwon6h1hmgh5nzi3j959v7rqq4ymd.lambda-url.us-east-1.localhost.localstack.cloud:4566\", \"content-length\": \"113\", \"accept\": \"application/json, text/plain, */*\", \"user-agent\": \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36\", \"content-type\": \"application/json\", \"origin\": \"http://localhost:5173\", \"referer\": \"http://localhost:5173/\", \"accept-encoding\": \"gzip, deflate\", \"accept-language\": \"en-US,en;q=0.9\", \"x-amzn-tls-cipher-suite\": \"ECDHE-RSA-AES128-GCM-SHA256\", \"x-amzn-tls-version\": \"TLSv1.2\", \"x-forwarded-proto\": \"http\", \"x-forwarded-for\": \"\", \"x-forwarded-port\": \"4566\"}, \"queryStringParameters\": {}, \"requestContext\": {\"accountId\": \"anonymous\", \"apiId\": \"xuimwon6h1hmgh5nzi3j959v7rqq4ymd\", \"domainName\": \"xuimwon6h1hmgh5nzi3j959v7rqq4ymd.lambda-url.us-east-1.localhost.localstack.cloud:4566\", \"domainPrefix\": \"xuimwon6h1hmgh5nzi3j959v7rqq4ymd\", \"http\": {\"method\": \"POST\", \"path\": \"/\", \"protocol\": \"HTTP/1.1\", \"sourceIp\": \"\", \"userAgent\": \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36\"}, \"requestId\": \"9e116416-d4ae-4f17-9f10-99a6c32ade29\", \"routeKey\": \"$default\", \"stage\": \"$default\", \"time\": \"07/May/2024:06:22:45 +0000\", \"timeEpoch\": 1715062965907}, \"body\": \"{\\\"userId\\\":\\\"\\\",\\\"rawPath\\\":\\\"42b81e16-8224-4a6a-afdd-0885f109f839/d9b2adcf-f5bd-4615-9a30-ceda43dbdd37/addresses.csv\\\"}\", \"isBase64Encoded\": false}", "trace-id": "Root=1-6639c8ba-de24a814914ccfd384221150;Parent=164797a13bc03142;Sampled=0"}' to executor 'ddacb0e247c671706a6212e98d410f01'
2024-05-07 15:22:50 2024-05-07T06:22:50.385 DEBUG --- [   asgi_gw_1] rolo.gateway.wsgi          : POST 172.19.0.6:4566/_localstack_lambda/ddacb0e247c671706a6212e98d410f01/invocations/44e27d76-a832-4f00-9070-519ab21e3cf8/logs
2024-05-07 15:22:50 2024-05-07T06:22:50.389  INFO --- [   asgi_gw_1] localstack.request.http    : POST /_localstack_lambda/ddacb0e247c671706a6212e98d410f01/invocations/44e27d76-a832-4f00-9070-519ab21e3cf8/logs => 202
2024-05-07 15:22:50 2024-05-07T06:22:50.391 DEBUG --- [   asgi_gw_0] rolo.gateway.wsgi          : POST 172.19.0.6:4566/_localstack_lambda/ddacb0e247c671706a6212e98d410f01/invocations/44e27d76-a832-4f00-9070-519ab21e3cf8/response
2024-05-07 15:22:50 2024-05-07T06:22:50.393  INFO --- [   asgi_gw_0] localstack.request.http    : POST /_localstack_lambda/ddacb0e247c671706a6212e98d410f01/invocations/44e27d76-a832-4f00-9070-519ab21e3cf8/response => 202
2024-05-07 15:22:50 2024-05-07T06:22:50.401 DEBUG --- [   asgi_gw_3] l.s.l.i.version_manager    : Got logs for invocation '44e27d76-a832-4f00-9070-519ab21e3cf8'
2024-05-07 15:22:50 2024-05-07T06:22:50.401 DEBUG --- [   asgi_gw_3] l.s.l.i.version_manager    : [presign-44e27d76-a832-4f00-9070-519ab21e3cf8] START RequestId: 44e27d76-a832-4f00-9070-519ab21e3cf8 Version: $LATEST
2024-05-07 15:22:50 2024-05-07T06:22:50.401 DEBUG --- [   asgi_gw_3] l.s.l.i.version_manager    : [presign-44e27d76-a832-4f00-9070-519ab21e3cf8] END RequestId: 44e27d76-a832-4f00-9070-519ab21e3cf8
2024-05-07 15:22:50 2024-05-07T06:22:50.401 DEBUG --- [   asgi_gw_3] l.s.l.i.version_manager    : [presign-44e27d76-a832-4f00-9070-519ab21e3cf8] REPORT RequestId: 44e27d76-a832-4f00-9070-519ab21e3cf8     Duration: 32.07 ms      Billed Duration: 33 ms  Memory Size: 128 MB     Max Memory Used: 128 MB
2024-05-07 15:22:50 2024-05-07T06:22:50.402  INFO --- [   asgi_gw_3] localstack.request.http    : POST / => 200
2024-05-07 15:22:50 2024-05-07T06:22:50.414 DEBUG --- [   asgi_gw_1] rolo.gateway.wsgi          : POST localhost:4566/
2024-05-07 15:22:50 2024-05-07T06:22:50.419 DEBUG --- [   asgi_gw_0] rolo.gateway.wsgi          : POST localhost:4566/
2024-05-07 15:22:50 2024-05-07T06:22:50.459 DEBUG --- [   asgi_gw_2] rolo.gateway.wsgi          : POST localhost:4566/
2024-05-07 15:22:50 2024-05-07T06:22:50.482 DEBUG --- [   asgi_gw_0] rolo.gateway.wsgi          : POST localhost:4566/
2024-05-07 15:22:50 2024-05-07T06:22:50.515 DEBUG --- [   asgi_gw_3] rolo.gateway.wsgi          : POST localhost.localstack.cloud:4566/user-files
2024-05-07 15:22:50 2024-05-07T06:22:50.598  INFO --- [   asgi_gw_3] localstack.request.aws     : AWS s3.PostObject => 204
2024-05-07 15:22:51 2024-05-07T06:22:51.423 DEBUG --- [ead-4 (_run)] localstack_persist.state   : Persisting state of services: ['logs', 'sts', 's3', 'cloudwatch']
2024-05-07 15:22:51 2024-05-07T06:22:51.424  INFO --- [ead-4 (_run)] localstack_persist.state   : Persisting state of service logs...
2024-05-07 15:22:51 2024-05-07T06:22:51.430 DEBUG --- [ead-4 (_run)] localstack_persist.state   : Finished persisting state of service logs
2024-05-07 15:22:51 2024-05-07T06:22:51.431  INFO --- [ead-4 (_run)] localstack_persist.state   : Persisting state of service sts...
2024-05-07 15:22:51 2024-05-07T06:22:51.434 DEBUG --- [ead-4 (_run)] localstack_persist.state   : Finished persisting state of service sts
2024-05-07 15:22:51 2024-05-07T06:22:51.434  INFO --- [ead-4 (_run)] localstack_persist.state   : Persisting state of service s3...
2024-05-07 15:22:51 2024-05-07T06:22:51.444 DEBUG --- [ead-4 (_run)] localstack_persist.state   : Finished persisting state of service s3
2024-05-07 15:22:51 2024-05-07T06:22:51.444  INFO --- [ead-4 (_run)] localstack_persist.state   : Persisting state of service cloudwatch...
2024-05-07 15:22:51 2024-05-07T06:22:51.450 DEBUG --- [ead-4 (_run)] localstack_persist.state   : Finished persisting state of service cloudwatch
2024-05-07 15:22:51 2024-05-07T06:22:51.450 DEBUG --- [ead-4 (_run)] localstack_persist.state   : Finished persisting 4 services.
2024-05-07 15:22:59 2024-05-07T06:22:59.134 DEBUG --- [   asgi_gw_1] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-05-07 15:23:01 2024-05-07T06:23:01.453 DEBUG --- [ead-4 (_run)] localstack_persist.state   : Nothing to persist - no services were changed
2024-05-07 15:23:04 2024-05-07T06:23:04.893 DEBUG --- [   asgi_gw_2] rolo.gateway.wsgi          : OPTIONS localhost.localstack.cloud:4566/_localstack/health
2024-05-07 15:23:04 2024-05-07T06:23:04.936 DEBUG --- [   asgi_gw_0] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-05-07 15:23:06 2024-05-07T06:23:06.873 DEBUG --- [   asgi_gw_3] rolo.gateway.wsgi          : OPTIONS localhost.localstack.cloud:4566/user-files?list-type=2&max-keys=1000&prefix=e4dd77d4-f358-4143-884b-ae1deaf00109%2F

However, if I upload another file to the same location, it will start to work, and every file uploaded to the same place afterwards also continues to work.

@brian-carnot
Copy link
Author

Also, I don't notice much for the cloudwatch logs, except I guess the failed upload only took 33 ms compared to the 240 ms of the completed one.

Screenshot 2024-05-07 at 15 31 30

@brian-carnot
Copy link
Author

Additionally, I checked the actual localstack persisted data. It seems like all of the CSVs get successfully uploaded, but there's something wrong with the file browser that prevents them from being seen.

I can't get or delete the "ghost" object though, so although it's present in the file system, it's not accessible by S3.

Screenshot 2024-05-07 at 15 33 58

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@GREsau @brian-carnot