release-6138

What's Changed

• Storage release 2024-07-23 by @problame in #8468

Full Changelog: release-6136...release-6138

release-6136

What's Changed

• tests: remove a leftover 'running' flag by @hlinnaka in #8216
• Restore running xacts from CLOG on replica startup by @knizhnik in #7288
• pageserver: add metric pageserver_secondary_resident_physical_size by @jcsp in #8204
• CI(build-tools): Remove libpq from build image by @bayandin in #8206
• fix(pageserver): include aux file in basebackup only once by @skyzh in #8207
• CI(gather-rust-build-stats): fix build with libpq by @bayandin in #8219
• tests: Make neon_xlogflush() flush all WAL, if you omit the LSN arg by @hlinnaka in #8215
• docs: Graceful storage controller cluster restarts RFC by @VladLazar in #7704
• Check status of connection after PQconnectStartParams by @knizhnik in #8210
• remote_storage config: move handling of empty inline table {} to callers by @problame in #8193
• pageserver: simpler, stricter config error handling by @jcsp in #8177
• fix: noisy logging when download gets cancelled during shutdown by @problame in #8224
• pageserver: rate limit log for loads of layers visited by @VladLazar in #8228
• Add support for reading and writing compressed blobs by @arpad-m in #8106
• L0 flush: opt-in mechanism to bypass PageCache reads and writes by @problame in #8190
• CI(benchmarking): move psql queries to actions/run-python-test-set by @bayandin in #8230
• tense of errors by @jcsp in #8234
• fix(pageserver): comments about metadata key range by @skyzh in #8236
• Add test for proper handling of connection failure to avoid 'cannot wait on socket event without a socket' error by @knizhnik in #8231
• docker: add storage_scrubber into the docker image by @skyzh in #8239
• Simplify test_wal_page_boundary_start test by @hlinnaka in #8214
• CI: update docker/* actions to latest versions by @bayandin in #7694
• pageserver: don't try to flush if shutdown during attach by @jcsp in #8235
• Only support compressed reads if the compression setting is present by @arpad-m in #8238
• add pagebench test cases for periodic pagebench on dedicated hardware by @Bodobolero in #8233
• pageserver: reduce ops tracked at per-timeline detail by @jcsp in #8245
• Fix outdated comment by @japinli in #8149
• page_service: stop exposing get_last_record_rlsn by @problame in #8244
• fix(pageserver): ensure test creates valid layer map by @skyzh in #8191
• pageserver: add supplementary branch usage stats by @jcsp in #8131
• tests: perform graceful rolling restarts in storcon scale test by @VladLazar in #8173
• proxy: cache certain non-retriable console errors for a short time by @conradludgate in #8201
• CI(build-and-test): add conclusion job by @bayandin in #8246
• pageserver: increase rate limit duration for layer visit log by @VladLazar in #8263
• Use bool param for round_trip_test_compressed by @arpad-m in #8252
• CI(pg-clients): unify workflow with build-and-test by @bayandin in #8160
• pageserver: downgrade stale generation messages to INFO by @jcsp in #8256
• Add find-large-objects subcommand to scrubber by @arpad-m in #8257
• feat(pageserver): integrate lsn lease into synthetic size by @yliang412 in #8220
• Flatten compression algorithm setting by @arpad-m in #8265
• implement rolling hyper-log-log algorithm by @knizhnik in #8068
• add checkout depth1 to workflow to access local github actions like generate allure report by @Bodobolero in #8259
• tests: extend allow list in deletion test by @jcsp in #8268
• correct error handling for periodic pagebench runner status by @Bodobolero in #8274
• tests: make location_conf_churn more robust by @jcsp in #8271
• safekeeper: add separate tombstones map for deleted timelines by @jcsp in #8253
• pageserver: add time based image layer creation check by @VladLazar in #8247
• SELECT 💣(); by @bayandin in #8270
• Increase timeout for wating subscriber caught-up by @knizhnik in #8118
• pageserver_live_connections: track as counter pair by @problame in #8227
• Improve parsing of ImageCompressionAlgorithm by @arpad-m in #8281
• Add concurrency to the find-large-objects scrubber subcommand by @arpad-m in #8291
• build(deps): bump certifi from 2023.7.22 to 2024.7.4 by @dependabot in #8301
• fix(pageserver): ensure sparse keyspace is ordered by @skyzh in #8285
• pageserver: switch to jemalloc by @jcsp in #8307
• Long-running replication tests by @save-buffer in #7896
• pageserver: respect has_relmap_file in collect_keyspace by @jcsp in #8276
• refactor: move part of sharding API from pageserver_api to utils by @yliang412 in #8254
• compute_ctl: Use 'fast' shutdown for Postgres termination by @ololobus in #8289
• tests: use smaller layers in test_pg_regress, do compaction after tests by @jcsp in #8232
• fix(pageserver): write to both v1+v2 for aux tenant import by @skyzh in #8316
• tests: stabilize test_timeline_size_quota_on_startup by @jcsp in #8255
• tests: increase approx size equal threshold to avoid test_lsn_lease_size flakiness by @yliang412 in #8282
• CI(promote-compatibility-data): take into account commit sha by @bayandin in #8283
• proxy/http: avoid spurious vector reallocations by @lucab in #8319
• chore(storage-scrubber): allow disable file logging by @skyzh in #8297
• chore: fix nightly build by @conradludgate in #8142
• fix(storage-scrubber): use default AWS authentication by @skyzh in #8299
• Refactor how migrations are ran by @tristan957 in #8326
• refactor: postgres_backend: replace abstract shutdown_watcher with CancellationToken by @problame in #8295
• fix(l0_flush): drops permit before fsync, potential cause for OOMs by @problame in #8327
• pageserver: move page_service's import basebackup / import wal to mgmt API by @problame in #8292
• proxy: pg17 fixes by @conradludgate in #8321
• tests: stabilize test_sharding_split_compaction by @jcsp in #8318
• proxy: remove some trace logs by @conradludgate in #8334
• build(deps-dev): bump zipp from 3.8.1 to 3.19.1 by @dependabot in #8329
• Remove ImageCompressionAlgorithm::DisabledNoDecompress by @arpad-m in #8300
• feat(pageserver): add k-merge layer iterator with lazy loading by @skyzh in #8053
• safekeeper: add eviction_min_resident to stop evictions thrashing by @jcsp in #8335
• Enable core dumps for postgres by @kelvich in #8272
• rfcs: add RFC for timeline archival by @jcsp in #8221
• Run Performance bench on more platforms by @Bodobolero in #8312
• pageserver: remove trace_read_requests by @problame in #8338
• storage_controller: fix ReconcilerWaiter::get_status by @VladLazar in #8341
• safekeeper: eviction metrics by @jcsp in #8348
• storage controller: add nod...

release-5940

What's Changed

• Storage & Compute release 2024-07-15 by @vipvap in #8374

Full Changelog: release-5889...release-5940

release-5889

What's Changed

• Storage & Compute release 2024-07-08 by @vipvap in #8304

Full Changelog: release-5837...release-5889

release-5837

What's Changed

• proxy: update tokio-postgres to allow arbitrary config params by @conradludgate in #8076
• Move remote_storage config related code into dedicated module by @arpad-m in #8132
• pageserver: remove code that resumes tenant deletions after restarts by @jcsp in #8091
• pageserver: add more info-level logging in shard splits by @jcsp in #8137
• tests: fix a flake in test_sharding_split_compaction by @jcsp in #8136
• Truncate waltmp file on creation by @petuhovskiy in #8133
• fix(pageserver): handle version number in draw timeline by @skyzh in #8102
• test(pageserver): add delta records tests for gc-compaction by @skyzh in #8078
• storcon_cli: remove old tenant-scatter command by @jcsp in #8127
• tests: accomodate some messages that can fail tests by @jcsp in #8144
• feat(pageserver): add an optional lease to the get_lsn_by_timestamp API by @yliang412 in #8104
• Fix MVCC bug with prepared xact with subxacts on standby by @hlinnaka in #8152
• storcon: update db related dependencides by @VladLazar in #8155
• L0 flush: avoid short-lived allocation when checking key_range empty by @problame in #8154
• CI: upload docker cache only from main branch by @bayandin in #8157
• feat(pageserver): add metrics for number of valid leases after each refresh by @yliang412 in #8147
• Fix submodule references to match the REL_*_STABLE_neon branches by @hlinnaka in #8159
• pageserver: remove attach/detach apis by @jcsp in #8134
• clippy-deny the todo!() macro by @problame in #4340
• proxy fix wake compute console retry by @conradludgate in #8141
• feat(pageserver): add image layer iterator by @skyzh in #8006
• bottom-most-compaction: use in test_gc_feedback + fix bugs by @skyzh in #8103
• add commit hash to S3 object identifier for artifacts on S3 by @Bodobolero in #8161
• Remove primary_is_running by @hlinnaka in #8162
• test(bottom-most-compaction): wal apply order by @skyzh in #8163
• Improve term reject message in walproposer by @petuhovskiy in #8164
• proxy: report blame for passthrough disconnect io errors by @conradludgate in #8170
• CI(build-tools): don't install Postgres 14 by @bayandin in #6540
• Panic on unexpected error in simtests by @petuhovskiy in #8169
• Silence compiler warning by @hlinnaka in #8153
• Add counters for commands processed through the libpq page service API by @hlinnaka in #8089
• docker: downgrade openssl to 1.1.1w by @VladLazar in #8168
• Evict WAL files from disk by @petuhovskiy in #8022
• pageserver: remove legacy tenant config code, clean up redundant generation none/broken usages by @jcsp in #7947
• test: use aux file v2 policy in benchmarks by @skyzh in #8174
• test: Add helper function for importing a Postgres cluster by @hlinnaka in #8025
• CI: additional trigger on merge to main by @fcdm in #8176
• strocon: don't overcommit when making node fill plan by @VladLazar in #8171
• Allow to change compute safekeeper list without restart. by @arssher in #8075
• CI: Use runner.arch in cache keys along with runner.os by @bayandin in #8175
• stocon: bump number of concurrent reconciles per operation by @VladLazar in #8179
• fix: shutdown does not kill walredo processes by @problame in #8150
• feat(pageserver): add delta layer iterator by @skyzh in #8064
• Improve slow operations observability in safekeepers by @petuhovskiy in #8188
• Add application_name to compute activity monitor connection string by @tristan957 in #8187
• pageserver: remove tenant create API by @jcsp in #8135
• virtual_file: take a Slice in the read APIs, eliminate read_exact_at_n, fix UB for engine std-fs by @problame in #8186
• Add buckets to safekeeper ops metrics by @petuhovskiy in #8194
• Cherry-pick upstream fix for TruncateMultiXact assertion by @hlinnaka in #8195
• pageserver: drop out of secondary download if iteration time has passed by @jcsp in #8198
• Add rate limiter for partial uploads by @petuhovskiy in #8203
• storage controller: fix heatmaps getting disabled during shard split by @jcsp in #8197
• fix(pageserver): ensure tenant harness has different names by @skyzh in #8205
• Fix tracking of the nextMulti in the pageserver's copy of CheckPoint by @knizhnik in #6528
• Storage & Compute release 2024-07-01 by @vipvap in #8217
• Rc/2024 07 01 without clog recovery by @arssher in #8269

Full Changelog: release-5779...release-5837

release-5779

What's Changed

• storcon: handle entire cluster going unavailable correctly by @VladLazar in #8060
• CI(check-build-tools-image): change build-tools image persistent tag by @bayandin in #8059
• Fix log rates by @MMeent in #8035
• [v2] Include openssl and ICU statically linked by @problame in #8074
• pageserver: eliminate CalculateSyntheticSizeError::LsnNotFound (test_metric_collection flake) by @jcsp in #8065
• safekeeper: streaming pull_timeline by @arssher in #7944
• Add support to specifying storage account in AzureConfig by @arpad-m in #8090
• build(deps): bump ws from 8.16.0 to 8.17.1 in /test_runner/pg_clients/typescript/serverless-driver by @dependabot in #8087
• build(deps): bump urllib3 from 1.26.18 to 1.26.19 by @dependabot in #8086
• Update document on Postgres core changes by @hlinnaka in #7416
• feat(pageserver): use leases to temporarily block gc by @yliang412 in #8084
• Remove dead code from walproposer_pg.c by @arssher in #8092
• feat(pageserver): add iterator API for btree reader by @skyzh in #8083
• Copy release images to prod ECR by @SergeyMelnikov in #8101
• storcon: add drain and fill background operations for graceful cluster restarts by @VladLazar in #8014
• test: fix tenant duplication utility generation numbers by @VladLazar in #8096
• neon_local: background_process: launch all processes in repo dir (or datadir) by @problame in #8058
• tests: make Endpoint.stop() thread safe (occasional flakes in test_multi_attach) by @jcsp in #8110
• proxy: reduce some per-task memory usage by @conradludgate in #8095
• allow storage_controller error during pagebench by @Bodobolero in #8109
• Make sure we can handle temporarily offline PS when we first connect by @MMeent in #8094
• tests: use semaphore instead of lock for Endpoint.running by @jcsp in #8112
• CI(test-images): add dockerhub auth by @bayandin in #8115
• CI: miscellaneous cleanups by @bayandin in #8073
• remove materialized page cache by @problame in #8105
• fix: preinitialize pageserver_basebackup_query_seconds metric by @problame in #8121
• storcon: use attached shard counts for initial shard placement by @VladLazar in #8061
• Add tracing for shared locks in id_lock_map by @jbajic in #7618
• storcon: avoid promoting too many shards of the same tenant by @VladLazar in #8099
• pageserver: remove unused load/ignore APIs by @jcsp in #8122
• Allow longer timeout for starting pageserver, safe keeper and storage controller in test cases to make test cases less flaky by @Bodobolero in #8079
• Add eviction_state to control file by @petuhovskiy in #8125
• pageserver: always detach before deleting by @jcsp in #8082
• storcon: implement endpoints for cancellation of drain and fill operations by @VladLazar in #8029
• tests: allow-list neon_local endpoint errors from storage controller by @jcsp in #8123
• Remove kq_imcx extension from public compute image by @lubennikovaav in #5201
• storcon: improve drain and fill shard placement by @VladLazar in #8119
• Use serde for RemoteStorageConfig parsing by @arpad-m in #8126
• Storage & Compute release 2024-06-24 by @vipvap in #8138

Full Changelog: release-5733...release-5779

release-5733

What's Changed

• test(pageserver): quantify compaction outcome by @skyzh in #7867
• Revert "Include openssl and ICU statically linked" by @problame in #8003
• Simplify scanning compute logs in tests by @hlinnaka in #7997
• fix: allow layer flushes more often by @koivunej in #7927
• docs: highlight neon env comes with an initial timeline by @skyzh in #7995
• refactor: Timeline layer flushing by @koivunej in #7993
• Add testing for extensions by @a-masterov in #7818
• fix: stop storing TimelineMetadata in index_part.json as bytes by @koivunej in #7699
• test: fix duplicated harness name by @koivunej in #8010
• feat(pageserver): initial code sketch & test case for combined gc+compaction at gc_horizon by @skyzh in #7948
• storcon: track number of attached shards for each node by @VladLazar in #8011
• storcon_cli: add 'drain' command by @VladLazar in #8007
• Copy editor config for the neon extension from PostgreSQL by @hlinnaka in #8009
• Rename S3 scrubber to storage scrubber by @arpad-m in #8013
• Add On-demand WAL Download to logicalfuncs by @save-buffer in #7960
• Another attempt at making test_vm_bits less flaky by @hlinnaka in #7989
• Update default Postgres version in docker-compose.yml by @hlinnaka in #8019
• Update documentation on running locally with Docker by @hlinnaka in #8020
• Resolve the problem the docker compose caused by the extensions tests by @a-masterov in #8024
• storcon_cli: do not drain to undesirable nodes by @VladLazar in #8027
• Reactivate page bench test in CI after ignoring CopyFail error in pageserver by @Bodobolero in #8023
• Add the image version to the neon-test-extensions image by @a-masterov in #8032
• test(pageserver): add test keyspace into collect_keyspace by @skyzh in #8016
• Fix query error in vm-image-spec.yaml by @MMeent in #8028
• Fix on-demand SLRU download on standby starting at WAL segment boundary by @hlinnaka in #8031
• Proxy process updated errors by @khanova in #8026
• test(pageserver): add test wal record for unit testing by @skyzh in #8015
• fix: vectored get returns incorrect result on inexact materialized page cache hit by @problame in #8050
• Set application_name for internal connections to computes by @tristan957 in #8047
• extensions: pgvector-0.7.2 by @jamesbroadhead in #8037
• pageserver: refine shutdown handling in secondary download by @jcsp in #8052
• Fix test_replica_query_race flakiness by @hlinnaka in #8038
• CI: Update outdated GitHub Actions by @bayandin in #8042
• storage controller: always wait for tenant detach before delete by @jcsp in #8049
• pageserver: improved synthetic size & find_gc_cutoff error handling by @jcsp in #8051
• chore: update to rust 1.79 by @conradludgate in #8048
• Fix test_segment_init_failure. by @arssher in #8056
• CI: downgrade docker/setup-buildx-action by @bayandin in #8062
• chore(pageserver): vectored get target_keyspace directly accums by @skyzh in #8055
• add halfvec indexing and queries to periodic pgvector performance tests by @Bodobolero in #8057
• Install rust binaries before running rust tests. by @arssher in #8054
• Release 2024-06-17 by @vipvap in #8069

New Contributors

• @jamesbroadhead made their first contribution in #8037

Full Changelog: release-5691...release-5733

release-5691

What's Changed

• test: allow no vectored gets happening by @koivunej in #7939
• feat(pageserver): collect aux file tombstones by @skyzh in #7900
• Move keyspace utils to inherent impls by @arpad-m in #7929
• s3_scrubber: add pageserver-physical-gc by @jcsp in #7925
• Store logical replication origin in KV storage by @knizhnik in #7099
• storcon_cli: add 'drop' and eviction interval utilities by @jcsp in #7938
• pageserver: don't squash all image layer generation errors into anyhow::Error by @jcsp in #7943
• [proxy] Fix dynamic rate limiter by @khanova in #7950
• Set page LSN when reconstructing VM in page server by @knizhnik in #7935
• test: no missed wakeups, cancellation and timeout flow to downloads by @koivunej in #7863
• build(Dockerfile.compute-node): do not log tar contents by @koivunej in #7953
• feat(rtc): maintain dirty and uploaded IndexPart by @koivunej in #7833
• pageserver: include heatmap in tenant deletion by @jcsp in #7928
• fix(pageserver): duplicate layers can cause corruption by @problame in #7799
• feat: support changing IndexPart::metadata_bytes to json in future release by @koivunej in #7693
• feat(pageserver): compute aux file size on initial logical size calculation by @skyzh in #7958
• upgrade pgvector from 0.7.0 to 0.7.1 by @Bodobolero in #7954
• chore!: remove walredo_process_kind config option & kind type by @problame in #7756
• pageserver/controller: enable tenant deletion without attachment by @jcsp in #7957
• chore(walredo): avoid duplicate tenant_id and shard_slug fields by @problame in #7977
• refactor(pageserver): designated api error type for cancelled request by @yliang412 in #7949
• fix: do TimelineMetrics::shutdown only once by @koivunej in #7983
• Perform retries on azure bulk deletion by @arpad-m in #7964
• fix(pageserver): more flexible layer size test by @skyzh in #7945
• fix(pageserver): skip metadata compaction if LSN is not accumulated enough by @skyzh in #7962
• raise the warning for oversized L0 to 2*target by @koivunej in #7985
• Remove random sleep in partial backup by @petuhovskiy in #7982
• Downgrade No response from reading prefetch entry WARNING to LOG by @lubennikovaav in #7988
• test_local_only_layers_after_crash: various fixes by @koivunej in #7986
• CI: copy run-* labels from external contributors' PRs by @a-masterov in #7915
• Move SQL migrations out of line by @tristan957 in #7991
• Include openssl and ICU statically linked by @rahulinux in #7956
• Release 2024-06-10 by @vipvap in #7999

New Contributors

• @yliang412 made their first contribution in #7949

Full Changelog: release-5657...release-5691

release-5657

What's Changed

• Fix connect to PS on MacOS/X by @knizhnik in #7885
• Add safekeeper test truncating WAL. by @arssher in #7891
• Performance test for pgvector HNSW index build and queries by @Bodobolero in #7873
• clarify how to load the dbpedia vector embeddings into a postgres database by @Bodobolero in #7894
• pageserver: fix a warning on secondary mode downloads after evictions by @jcsp in #7877
• Drop postgres-native-tls in favour of tokio-postgres-rustls by @arpad-m in #7883
• proxy: reintroduce dynamic limiter for compute lock by @conradludgate in #7737
• Do not produce error if gin page is not restored in redo by @knizhnik in #7876
• CI: switch ubuntu-latest with ubuntu-22.04 (#7256) by @a-masterov in #7901
• proxy fix wake compute rate limit by @conradludgate in #7902
• proxy: upload postgres connection options as json in the parquet upload by @conradludgate in #7903
• proxy: lazily parse startup pg params by @conradludgate in #7905
• refacter : VirtualFile::open uses AsRef by @SeinoYuki in #7908
• fix(pageserver): ensure to_i128 works for metadata keys by @skyzh in #7895
• perf(pageserver): postpone vectored get fringe keyspace construction by @skyzh in #7904
• chore(pageserver): warn on delete non-existing file by @skyzh in #7847
• Update tokio-epoll-uring for linux-raw-sys by @arpad-m in #7918
• pageserver: quieten some shutdown logs around logical size and flush by @jcsp in #7907
• neon_walreader: check after local read that the segment still exists. by @arssher in #7898
• Add test checking term change during pull_timeline by @arssher in #7893
• safekeeper: rename epoch to last_log_term. by @arssher in #7892
• pageserver: avoid spurious "bad state" logs/errors during shutdown by @jcsp in #7912
• Add FullAccessTimeline guard in safekeepers by @petuhovskiy in #7887
• test(pageserver): add test interface to create artificial layers by @skyzh in #7899
• pageserver: remove AncestorStopping error variants by @jcsp in #7916
• pagectl: key command for dumping what we know about the key by @koivunej in #7890
• pageserver: add GcError type by @jcsp in #7917
• Run gc on too many partial backup segments by @petuhovskiy in #7700
• Fix span for WAL removal task by @petuhovskiy in #7930
• Add metrics for Azure blob storage by @arpad-m in #7933
• Release 2024-06-03 by @vipvap in #7936

New Contributors

• @a-masterov made their first contribution in #7901
• @SeinoYuki made their first contribution in #7908

Full Changelog: release-5625...release-5657

release-5625

What's Changed

• Compute release 2024-05-31 by @bayandin in #7920

Full Changelog: release-5623...release-5625