[Bug]: promtail에서 loki로 로그를 보낼 때 loki에서 400을 반환하는 오류

[goathoon]

Describe the bug

level=error ts=2024-01-19T09:46:28.758744044Z caller=client.go:430 component=client host=115.145.160.240:3101 msg="final error sending batch" status=400 tenant= error="server returned HTTP status 400 Bad Request (400): entry with timestamp 2024-01-18 21:45:57.948654186 +0000 UTC ignored, reason: 'entry too far behind, oldest acceptable timestamp is: 2024-01-19T08:46:25Z',"

loki의 기본 설정 값 (limits_config.oldest acceptable timestamp) 이 1주일로 설정되어, 1주일보다 오래된 log를 loki에게 전송시 400 error를 반환합니다.

Reproduction

Logs

level=error ts=2024-01-19T09:46:28.758744044Z caller=client.go:430 component=client host=115.145.160.240:3101 msg="final error sending batch" status=400 tenant= error="server returned HTTP status 400 Bad Request (400): entry with timestamp 2024-01-18 21:45:57.948654186 +0000 UTC ignored, reason: 'entry too far behind, oldest acceptable timestamp is: 2024-01-19T08:46:25Z',

System Info

No response

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines
• Check that there isn't already an issue