Releases: yugabyte/yugabyte-db
v2.3.0.0 (Released September 8, 2020)
YSQL
- Fix OOM when running large
COPY TO
statements by creating a new memory context for the loop over retrieved rows and resetting it after processing each row. #5205 - Fix OOM on file-sourced YB relations in the
COPY FROM
statement. Reset memory context regularly, per row, when memory is resettable and when rows are read from a file (notstdin
). #5561 - Support transactional batch size for
COPY FROM
command with OOM fix. Batch sizes can be passed in withROWS_PER_TRANSACTION
in theCOPY OPTION
syntax. #2855 #5453 - For index backfill flags, use better default values. Set
index_backfill_rpc_timeout_ms
default from60000
to30000
and changebackfill_index_timeout_grace_margin_ms
default from50
to500
. #5494 - Remove spurious error message "0A000: Alter table is not yet supported" from
CREATE OR REPLACE VIEW
. #5071 - Prevent consistency violations when a partitioned table has foreign key constraints due to erroneous classification as a single-row transaction. #5387
- Fix restore from a distributed backup fails for tables using
SPLIT INTO
without a primary key. #4993 - Fix the wrong result by avoiding pushdown of
UPDATE
statement with theRETURNING
clause. #5366 - Improve error message when
UPDATE
changes partition so its clear what went wrong, fixedyb_pg_foreign_key
pg_regress
test for semantic merge conflict when updating primary keys and row-level partitioning. #659 #5179 #5310 - Allow the UPDATE statement to change primary key columns. #659
- Support row-level partitioning. #5179
- Block usage of
TABLEGROUP
with theSPLIT
clause. ForCREATE TABLE
, usage is blocked in the grammar. ForCREATE INDEX
, ifNO TABLEGROUP
was provided, then presplitting for the index is allowed; otherwise, an error is issued. #5352 - Buffered operations may share single RPC with read operation. Reducing the number of RPC calls speeds up the
CREATE TABLE
statement. For example, the total number of RPC calls is dropped from 66 to 58 for a simple table likeCREATE TABLE t(k INT PRIMARY KEY)
. #5177 - Add new
ysqlsh
describe metacommands for tablegroups:\dgr[+] [grpname]
to describe tablegroups,\dgrt[+] [grpname]
lists all tables/indexes within the specified tablegroup (or within all tablegroups ifgrpname
is not specified), and\d <table_name>
is modified to include tablegroup information in the footer, if any. #5088 - Add
ALTER TABLEGROUP
statements to supportALTER TABLEGROUP tablegroup_name RENAME TO ...
andALTER TABLEGROUP tablegroup_name OWNER TO ...
. Also changespg_tablegroup
entry corresponding totablegroup_name
to properly reflect newgrpname
andgrpowner
(if the user has proper permissions or ownership to issue theALTER TABLEGROUP
statement. #5249 - Add support for indexes to opt out of tablegroups (
NO TABLEGROUPS
) or select their own tablegroup (TABLEGROUP group_name
). #5293 - For
ysql_dump
, enableserializable-deferrable
mode by default. Add new--no-serializable-deferrable
flag to disable the default mode. #5128 - Support
ALTER COLUMN type
for variants that don't require on-disk changes. Specifically only allowingALTER COLUMN type
forvarch(n)
andvarbit(n)
. For example, changing column type fromvarchar(50)
tovarchar(255)
. #4424 - Enable
USING
clause inDELETE
statement andFROM
clause inUPDATE
statement. #738 #5262 - Support GRANT/REVOKE/ALTER DEFAULT PRIVILEGES for tablegroups. #5087 #5160
- Implement tablegroup query layer changes. #4525
- Replace "Foreign Scan" with "Seq Scan" for
EXPLAIN
statement. #2076 - Fix
initdb
when index backfill is enabled. #5027 - Properly handle empty delete with backfill by sending appropriate messages to the client, including setting it
skipped
androws_affected_count
to0
. #5015 DROP INDEX
statement invalidates table cache entry for the index table but should invalidate the table cache entry for the indexed table. #4974yb-admin create_database_snapshot
command should not requireysql.
prefix for database name. #4991- For non-prepared statements, optimize
pg_statistic
system table lookups and update debugging utilities. #5051 - Correctly show beta feature warnings by default. #5322
YCQL
- For
WHERE
clause inCREATE INDEX
statement, return aNot supported
error. #5363 - Fix TSAN issue in partition-aware policy for C++ driver 2.9.0-yb-8 (yugabyte/cassandra-cpp-driver). #1837
- Support YCQL backup for indexes based on JSON-attribute. #5198
- Correctly set
release_version
forsystem.peers
queries. #5407 - Reject
TRUNCATE
operations whenycql_require_drop_privs_for_truncate
flag is enabled. When enabled,DROP TABLE
permission is required to truncate a table. Default isfalse
. #5443 - Fix missing return statement in error case of the
SetPagingState
method instatement_params.cc
. #5441 - Enable backfilling of transactional tables by default. #4708
- Fix
ycqlsh
should return failure when known that the create (unique) index has failed. #5161
Core database
- Fix core dump related to DNS resolution from cache for Kubernetes universes. #5561
- Fix yb-master fails to restart after errors on the first run. #5276
- Show better error message when using
yugabyted
and yb-master fails to start. #5304 - Disable ignoring deleted tablets on load by default. #5122
- [CDC] Improve CDC idle throttling logic to reduce high CPU utilization in clusters without workloads running. #5472
- For
server_broadcast_addresses
flag, provide default port if not specified. #2540 - [CDC] Fix CDC TSAN destructor warning. #4258
- Add API endpoint to download the root certificate file. #4957
- Do not load deleted tables and tablets into memory on startup. #5122
- Set the follower lag for leaders to
0
by resetting timestamp to the maximum value when a peer becomes a leader and when the peer loses leadership. #5502 - Flow keyspace information from yb-master to yb-tserver. #3020
- Implement meta cache lookups throttling to reduce unnecessary thrashing. #5434
- Fix
rpcz/statements
links inyb-tserver
Web UI whenpgsql_proxy_bind_address
andcql_proxy_bind_address
are0.0.0.0
. #4963 DumpReplayStateToStrings
should handle too many WAL entries scenario. Also, log lines only display fields critical for debugging and do not show customer-sensitive information. #5345- Find the difference between the replica map and consensus state more quickly using map lookup. #5435
- Improve failover to a new master leader in the case of a network partition or a dead yb-tserver by ...
v2.2.2.0 (Released August 19, 2020)
YSQL
- Fix failed backup if restored table was deleted before restoration. #5274
- Newly elected YB-Master leader should pause before initiating load balancing. #5221
- Fix backfilling to better handle large indexed table tablets. #5031
- Fix
DROP DATABASE
statement should work with databases deleted on YB-Master. #4710 - For non-prepared statements, optimize
pg_statistic
system table lookups. #5051 - [CDC] Avoid periodic querying of the
cdc_state
table for xDC metrics if there are no replication streams enabled. #5173
YCQL
- Implement DNS cache to significantly reduce CPU loads due to a large number of DNS resolution requests (especially for YCQL connections). Adds
dns_cache_expiration_ms
flag (default is 1 minute). #5201 - Fixed incorrect names de-mangling in index creation from
CatalogManager::ImportSnapshot()
. #5157 - Fixed crashes when inserting literals containing newline characters. #5270
- Reuse CQL parser between processors to improve memory usage. Add a new
cql_processors_limit
flag to control processor allocation. #5057
Core database
- Fix
yugabyted
fails to start UI due to class binding failure. #5069 - Show hostnames in YB-Master and YB-TServer Admin UI when hostnames are specified in
--webserver_interface
,--rpc_bind_addresses
, and--server_broadcast_addresses
flags. #5002 - Skip tablets without intents during commits, for example, the case of an update of a non-existing row. #5321
- Fix log spew when applying an unknown transaction and release a mutex as soon as possible when the transaction is not found. #5315
- Fix snapshots cleanup when snapshots removed before restart. #5337
- Replace
SCHECK
withLOG(DFATAL)
when checking for restart-safe timestamps in WAL entries. #5314 - Replace all CHECK in the load balancer code with
SCHECK
orRETURN_NOT_OK
. #5182 - Implement DNS cache to significantly reduce CPU loads due to a large number of DNS resolution requests (especially for YCQL connections). Adds
dns_cache_expiration_ms
flag (default is 1 minute). #5201 - Avoid duplicate DNS requests when handling
system.partitions
table requests. #5225 - Fix leader load balancing can cause
CHECK
failures if stepdown task is pending on the next run. Sets the global leader balance threshold while allowing progress to be made across tables. Adds newload_balancer_max_concurrent_moves_per_table
flag to limit number of leader moves per table. #5181 and #5021 - Set the async
YBClient
initialization future inTabletPeer
constructor to ensure tablet bootstrap logic can resolve transaction statuses. #5215 - Handle write operation failures during tablet bootstrap. #5224
- Drop index upon failed backfill. #5144 #5161
- Fix WAL overwriting by new leader and replay of incorrect entries on tablet bootstrap. #5003 #3759 #4983
- Avoid taking a unique lock when starting a lookup request. #5059
- Set 2DC lag metrics to
0
if not the leader and if the replication is deleted. #5113 - Set the table to ALTERING state when
fully_*
is populated. #5139 Not the leader
errors should not cause a replica to be marked as failed. #5072- Use the difference between follower's hybrid time and its safe time as a measure of staleness. #4868
Yugabyte Platform
- Add Master section below Tablet Server section in Metrics page. #5233
- Add
rpc_connections_alive
metrics for YSQL and YCQL APIs. #5223 - Fix restore payload when renaming table to include keyspace. Disable keyspace field when restoring universe backup. #5178
- Pass in
ssh_user
to air-gap provision script and add to the on-premise template. #5132 - Add the
--recursive
flag to AZCopy for multi-table restore. #5163 - Fix transactional backup with specified tables list by including keyspace in payload. #5149
- Fix undefine object property error when Prometheus is unavailable and navigating to the Replication tab. #5146
- Backups should take provider-level environment variables, including home directory. #5064
- Add hostname and display in the Nodes page along with node name and IP address. #4760
- Support the option of DNS names instead of IP addresses for nodes. Add option in Admin UI to choose between using hostnames or IP addresses for the on-premises provider. #4951 #4950
- Disable glob before running cleanup of old log files using
zip_purge_yb_logs.sh
. Fixes issue on Red Hat. #5169 - Fix Replication graph units and missing graph in Replication tab when metrics exist. #5423
Note:
Prior to version 2.0, YSQL was still in beta. As a result, the 2.0 release included a backward-incompatible file format change for YSQL. If you have an existing cluster running releases earlier than 2.0 with YSQL enabled, then you will not be able to upgrade to version 2.0+. Export from your existing cluster and then import into a new cluster (v2.0 or later) to use existing data.
v2.2.0.0 (Released July 15, 2020)
YSQL
Transactional distributed backups
YugabyteDB now supports distributed backup and restore of YSQL databases, including backup of all tables in a database. #1139 and #3849
Online index backfills
- YugabyteDB can now build indexes on non-empty tables while online, without failing other concurrent writes. When you add a new index to a table that is already populated with data, you can now use the YSQL
CREATE INDEX
statement to enable building these indexes in an online manner, without requiring downtime. For details on how the online backfill of indexes works, see the Online Index Backfill design document. - Backfilling an index while online is disable by default. To enable online index backfilling, set the
yb-tserver
--ysql_disable_index_backfill
flag tofalse
when starting YB-TServers. For details on how this works, see Online Index Backfill.
Colocated tables
Database-level colocation for YSQL, which started as a beta feature in the 2.1 release, is now generally available in the 2.2 release. Traditional RDBMS modeling of parent-child relationships as foreign key constraints can lead to high-latency JOIN
queries in a geo-distributed SQL database. This is because the tablets (or shards) containing the child rows might be hosted in nodes, availability zones, and regions different from the tablets containing the parent rows. By using colocated tables, you can let a single tablet be shared across all tables. Colocation can also be at the overall database level, where all tables of a single database are located in the same tablet and managed by the same Raft group. Note that tables that you do not want to reside in the overall database’s tablet because of the expectation of large data volume can override the feature at table creation time and hence get independent tablets for themselves.
What happens when the “colocation” tablet containing all the tables of a database becomes too large and starts impacting performance? Check out automatic tablet splitting [BETA].
Deferred constraints on foreign keys
Foreign keys in YSQL now support the DEFERRABLE INITIALLY IMMEDIATE and DEFERRABLE INITIALLY DEFERRED clauses. Work on deferring additional constraints, including those for primary keys, is in progress.
Application developers often declare constraints that their data must obey, leaving it to relational databases to enforce the rules. The end result is simpler application logic, lower error probability, and higher developer productivity. Automatic constraint enforcement is a powerful feature that should be leveraged whenever possible. There are times, however, when you need to temporarily defer enforcement. An example is during the data load of a relational schema where there are cyclic foreign key dependencies. Data migration tools usually defer the enforcement of foreign key dependencies to the end of a transaction by which data for all foreign keys would ideally be present. This should also allow YSQL to power Django apps.
yugabyted for single-node clusters
The yugabyted
server is now out of beta for single-node deployments. New users can start using YugabyteDB without needing to understand the underlying architectures acts as a parent server, reducing the need to understand data management by YB-TServers and metadata management by YB-Masters.
See it in action by following the updated Quick start. For details, see yugabyted
in the Reference section.
Automatic tablet splitting [BETA]
- YugabyteDB now supports automatic tablet splitting, resharding your data by changing the number of tablets at runtime. For details, see Tablet splitting and Automatic Resharding of Data with Tablet Splitting. #1004, #1461, and #1462
- To enable automatic tablet splitting, use the new
yb-master
--tablet_split_size_threshold_bytes
flag to specify the size when tablets should split.
TPC-C benchmarking
New results are now available for benchmarking the performance of the YSQL API using the TPC-C suite. For the new TPC-C results and details on performing your own benchmark tests to evaluate YugabyteDB, see TPC-C.
Other notable changes
- Support pre-splitting using CREATE INDEX...SPLIT INTO for range-partitioned table indexes. For details, see Pre-splitting #4235
- Fix crash for nested
SELECT
statements that involve null pushdown on system tables. #4685 - Fix wrong sorting order in pre-split tables. #4651
- To help track down unoptimized (or "slow") queries, use the new
yb-tserver
--ysql_log_min_duration_statement
. #4817 - Enhance the
yb-admin list_tables
command with optional flags for listing tables with database type (include_db_type
), table ID (include_table_id
), and table type (include_table_type
). This command replaces the deprecatedyb-admin list_tables_with_db_types
command. #4546 - Add support for
ALTER TABLE
on colocated tables. #4293 - Improve logic for index delete permissions. #4980
- Add fast path option for index backfill when certain statements can create indexes without the unnecessary overhead from online schema migration (for example,
CREATE TABLE
with unique column constraint). Skip index backfill for unsupported statements, includingDROP INDEX
,CREATE UNIQUE INDEX ON
, and index create in postgres nested DDL. [#4918] - Suppress
incomplete startup packet
messages in YSQL logs. #4813 - Improve YSQL create namespace failure handling. #3979
- Add error messages to table schema version mismatch errors so they are more understandable. [#4810]
- Increase the default DDL operations timeout to 120 seconds to allow for multi-region deployments, where a CREATE DATABASE statement can take longer than one minute. #4762
YCQL
Transactional distributed backups
YugabyteDB supports distributed backup and restore of YCQL databases and tables. #1139 and #3849
Online index backfills
- YugabyteDB can now build indexes on non-empty tables while online, without failing other concurrent writes. When you add a new index to a table that is already populated with data, you can now use the YCQL CREATE INDEX statement to enable building these indexes in an online manner, without requiring downtime. For details on how the online backfill of indexes works, see the Online Index Backfill design document.
- In YCQL, backfilling an index while online is enabled by default. To disable, set the
yb-tserver
--ycql_disable_index_backfill
flag tofalse
when starting YB-TServers. Note: Do not use this flag in a production cluster yet. For details on how online index backfill works, see Online Index Backfill #2301 and #4708
Online schema changes for YCQL [BETA]
Most applications have a need to frequently evolve the database schema, while simultaneously ensuring zero downtime during those schema change ope...
v2.1.8.2 (Released June 19, 2020)
YSQL
- Add support for
SPLIT INTO
clause forCREATE INDEX
statement. #3047 - Fix aggregate functions pushdown for columns with default values added after table creation. #4376
- Resolve timeout frequently encountered whe batch-loading data in YSQL by using client-specified timeouts for RPC requests instead of hardcoded values. #4045
- Fix incorrect cross-component dependency in DocDB found in builds using
ninja
. #4474 - Fix operation buffering in stored procedures to handle transactions correctly. #4268
- [DocDB] Ensure that fast path (pushed down) single row writes honor higher priority transactions and get aborted or retired instead. #4316
- [DocDB] Split copy table operations into smaller chunks (using byte size instead of count) for CREATE DATABASE statement. #3743
- Correctly push down
IS NULL
condition to DocDB. #4499 - Avoid ASAN failures after a large data set is uploaded by rearranging files and test functionalities. #4488
- Fix memory leaks by using memory context to manage object alloc and free. #4490
- Fix multi-touch cache and improve caching logic for read and write operations. #4379
- Improve index cost estimates by considering index uniqueness, included columns (index scan vs. index only scan), scan direction, and partial indexes. Also, disable merge joins for unsupported cases. #4494 and #4496
- Add support for deferrable foreign key constraints. #3995
- Prevent dropping primary key constraint. #3163
- Push down
SELECT <aggregate>(<const>)
, for example,SELECT COUNT(1)
, to DocDB. #4276 - Fix rare core dumps due to concurrency issues in metrics webserver during shutdown. #4092
YCQL
- Rename
cqlsh
toycqlsh
. #3935 - Update Cassandra Java driver version to
3.8.0-yb-4
and adds support forguava
26 or later. The latest release of the driver is available in the Yugabytecassandra-java-driver
repository. #3897 - YB-TServers should not crash when attempting to log in using YCQL authentication without a password. #4459
- Performance degradation in
CassandraSecondaryIndex
workload resolved. #4401 - Add
yb-admin import_snapshot
support for renaming a few tables (not all), but the specified name is equal to the old table name:yb-admin import_snapshot <meta-file> ks old_table_name
. #4280 - For DDL creation with Spring Data Cassandra, change the
Enum
value fromJSON
toJSONB
to allow schema creation to succeed programmatically involving JSON column types and update thecassandra-java-driver
to3.8.0-yb-5
. #4481 - Use the same timestamp for current time to compute multiple runtimes in output of
<tserver-ip>:13000/rpcz
. #4418 - Correctly push down
= NULL
condition to DocDB. #4499 - Reduce YCQL unprepared statement execution time by up to 98% (example: reduced time to insert a 5 MB string from 18 seconds to 0.25 seconds). #4397 and #3586
- Special thanks to @ouvai59 for your contribution!
YEDIS
- For
yugabyted
, do not startredis
server by default. Resolves port conflict during startup. #4057
System improvements
- New
yb-admin
commandget_load_balancer_state
to get the cluster load balancer state. #4509 - Avoid creating intent iterator when no transactions are running. #4500
- Increase the default memory limit for
yb-master
for running in low-memory setups (<=4 GB
). #3742 - Improve RocksDB checkpoint directory cleanup if a
yb-tserver
crashes or is restarted while performing a snapshot operation. #4352 - [DocDB] Use bloom filters for range-partitioned tables. The first primary key column is added to the bloom filter. #4437
- [DocDB] Fix snapshots bootstrap order bu altering the load for transaction-aware snapshots. #4470
- [DocDB] Add
yb-admin master_leader_stepdown
command. #4135 - [DocDB] Reduce impact on CPU and throughput during node failures. #4042
- [DocDB] Add
yb-ts-cli
commands,flush_all_tablets
andflush_tablet <tablet_id>
, to flush tablets. When used with rolling restarts, less time is spent applying WAL records to rocksdb. #2785- Special thanks to mirageyjd for your contribution.
- [DocDB] Fix deadlock during tablet splitting. #4312
- Introduced load balancing throttle on the total number of tablets being remote bootstrapped, across the cluster. #4053 and #4436
- [DocDB] Remove applied intent doc hybrid time during compaction. #4535
- [DocDB] Fixed BoundedRocksDbIterator::SeekToLast works incorrectly for 2nd post-split tablet. #4542
- [DocDB] Abort snapshot if table was deleted. #4610
- [DocDB] Backfill index without waiting indefinitely for pending transactions. #3471
- [DocDB] Fix
yb-master
rerunning snapshot operations after upgrade. #4816 - [Colocation] During load balancing operations, load balance each colocated tablet once. This fix removes unnecessary load balancing for every user table sharing that table and the parent table.
- Fix YB-Master hangs due to transaction status resolution. #4410
- Redirect the master UI to the master leader UI without failing when one master is down. #4442 and #3869
- Avoid race in
change_metadata_operation
. Use atomic<P*> to avoid race between
Finish()
andToString
from updating or accessing request. #3912 - Refactor
RaftGroupMetadata
to avoid keeping unnecessaryTableInfo
objects in memory. #4354 - Fix missing rows in unidirectional replication and fix race conditions with CDC and TransactionManager. #4257
- Change intent iterator creation logic. #4543
- Upgrade all Python scripts used to build and package the code to Python 3. #1442
Yugabyte Platform
- Fix failure when adding a node on a TLS-enabled universe. #4482
- Improve latency tracking by splitting overall operation metrics into individual rows for each API. #3825
- YCQL and YEDIS metrics include
ops
,avg latency
, andP99 latency
. - YSQL metrics include only
ops
andavg latency
.
- YCQL and YEDIS metrics include
- Add metrics for RPC queue sizes of services, including YB-Master, YB-TServer, YCQL, and YEDIS. #4294
- Add the option to edit configuration flags without requiring a server restart. #4433
- When configuration flags are deleted in the YugabyteDB Admin Console, remove the flags from
server.conf
file. [#4341](https://github.com/yugabyte/yugaby...
v2.1.8.1 (June 16, 2020)
YSQL
- Add support for
SPLIT INTO
clause forCREATE INDEX
statement. #3047 - Fix aggregate functions pushdown for columns with default values added after table creation. #4376
- Resolve timeout frequently encountered whe batch-loading data in YSQL by using client-specified timeouts for RPC requests instead of hardcoded values. #4045
- Fix incorrect cross-component dependency in DocDB found in builds using
ninja
. #4474 - Fix operation buffering in stored procedures to handle transactions correctly. #4268
- [DocDB] Ensure that fast path (pushed down) single row writes honor higher priority transactions and get aborted or retired instead. #4316
- [DocDB] Split copy table operations into smaller chunks (using byte size instead of count) for CREATE DATABASE statement. #3743
- Correctly push down
IS NULL
condition to DocDB. #4499 - Avoid ASAN failures after a large data set is uploaded by rearranging files and test functionalities. #4488
- Fix memory leaks by using memory context to manage object alloc and free. #4490
- Fix multi-touch cache and improve caching logic for read and write operations. #4379
- Improve index cost estimates by considering index uniqueness, included columns (index scan vs. index only scan), scan direction, and partial indexes. Also, disable merge joins for unsupported cases. #4494 and #4496
- Add support for deferrable foreign key constraints. #3995
- Prevent dropping primary key constraint. #3163
- Push down SELECT (), for example, SELECT COUNT(1) , to DocDB. #4276
- Fix rare core dumps due to concurrency issues in metrics webserver during shutdown. #4092
YCQL
- Rename
cqlsh
toycqlsh
. #3935 - Update Cassandra Java driver version to
3.8.0-yb-4
and adds support forguava
26 or later. The latest release of the driver is available in the Yugabytecassandra-java-driver
repository. #3897 - YB-TServers should not crash when attempting to log in using YCQL authentication without a password. #4459
- Performance degradation in
CassandraSecondaryIndex
workload resolved. #4401 - Add
yb-admin import_snapshot
support for renaming a few tables (not all), but the specified name is equal to the old table name:yb-admin import_snapshot <meta-file> ks old_table_name
. #4280 - For DDL creation with Spring Data Cassandra, change the
Enum
value fromJSON
toJSONB
to allow schema creation to succeed programmatically involving JSON column types and update thecassandra-java-driver
to3.8.0-yb-5
. #4481 - Use the same timestamp for current time to compute multiple runtimes in output of
<tserver-ip>:13000/rpcz
. #4418 - Correctly push down
= NULL
condition to DocDB. #4499 - Reduce YCQL unprepared statement execution time by up to 98% (example: reduced time to insert a 5 MB string from 18 seconds to 0.25 seconds). #4397 and #3586
- Special thanks to @ouvai59 for your contribution!
YEDIS
- For
yugabyted
, do not startredis
server by default. Resolves port conflict during startup. #4057
System improvements
- New
yb-admin
commandget_load_balancer_state
to get the cluster load balancer state. #4509 - Avoid creating intent iterator when no transactions are running. #4500
- Increase default memory limit for
yb-master
for running in low-memory setups (<=4 GB
). #3742 - Improve RocksDB checkpoint directory cleanup if a tserver crashes or is restarted while performing a snapshot operation. #4352
- [DocDB] Use bloom filters for range-partitioned tables. The first primary key column is added to the bloom filter. #4437
- [DocDB] Fix snapshots bootstrap order bu altering the load for transaction-aware snapshots. #4470
- [DocDB] Add
yb-admin master_leader_stepdown
command. #4135 - [DocDB] Reduce impact on CPU and throughput during node failures. #4042
- [DocDB] Add
yb-ts-cli
commands,flush_all_tablets
andflush_tablet <tablet_id>
, to flush tablets. When used with rolling restarts, less time is spent applying WAL records to rocksdb. #2785- Special thanks to mirageyjd for your contribution.
- [DocDB] Fix deadlock during tablet splitting. #4312
- Introduced load balancing throttle on the total number of tablets being remote bootstrapped, across the cluster. #4053 and #4436
- [DocDB] Remove applied intent doc hybrid time during compaction. #4535
- [DocDB] Fixed BoundedRocksDbIterator::SeekToLast works incorrectly for 2nd post-split tablet. #4542
- [DocDB] Abort snapshot if table was deleted. #4610
- [DocDB] Backfill index without waiting indefinitely for pending transactions. #3471
- [Colocation] During load balancing operations, load balance each colocated tablet once. This fix removes unnecessary load balancing for every user table sharing that table and the parent table.
- Fix YB-Master hangs due to transaction status resolution. #4410
- Redirect the master UI to the master leader UI without failing when one master is down. #4442 and #3869
- Avoid race in
change_metadata_operation
. Use atomic<P*> to avoid race between
Finish()
andToString
from updating or accessing request. #3912 - Refactor
RaftGroupMetadata
to avoid keeping unnecessaryTableInfo
objects in memory. #4354 - Fix missing rows in unidirectional replication and fix race conditions with CDC and TransactionManager. #4257
- Change intent iterator creation logic. #4543
- Upgrade all Python scripts used to build and package the code to Python 3. #1442
Yugabyte Platform
- Fix failure when adding a node on a TLS-enabled universe. #4482
- Improve latency tracking by splitting overall operation metrics into individual rows for each API. #3825
- YCQL and YEDIS metrics include
ops
,avg latency
, andP99 latency
. - YSQL metrics include only
ops
andavg latency
.
- YCQL and YEDIS metrics include
- Add metrics for RPC queue sizes of services, including YB-Master, YB-TServer, YCQL, and YEDIS. #4294
- Add option to edit configuration flags without requiring server restart. #4433
- When configuration flags are deleted in the YugabyteDB Admin Console, remove the flags from
server.conf
file. #4341 - When creating GCP instances, only use host project when specifying network.
- When creating a cloud provider configuration, display provider-level (non-k8s) settings for SSH ports and enabling airgapped installations. #3615, [#4243](ht...
v2.1.6 (May 8, 2020)
YSQL
- Wait for
tserver
to finish creating thetransaction
table during the initial cluster startup (when the transaction table is first created) before before issuing requests that require it to exist. This was more likely an issue for CI/CD, where requests can be issued immediately. Most users would not encounter this issue. #4056 - Avoid redundant read for non-unique index inserts. For non-unique indexes, the primary key of the main table is implicitly added to the DocDB key, guaranteeing uniqueness of the full DocDB key (indexed columns plus encoded base table primary key). This fix executes such inserts as upserts and avoid the read and uniqueness check. #4363
- Enhance automatic query read restart to avoid recreating portal. Instead of recreating a portal, reset an existing one to the state which allows it to be re-executed. Eliminate memory overhead for storing potential big bind variable values (for example, long strings). #4254
- For
CREATE DATABASE
statements, improves fault tolerance by making CREATE API requests asynchronously and adds a state machine on namespaces to be the authority for processing these modifications. #3097 - Display current query runtime (
process_running_for_ms
), in milliseconds (ms), on<tserver_ip>:13000/rpcz
endpoint. #4382
YCQL
- Allow
system.peers_v2
table to be readable forcassandra-driver-core:3.8.0-yb-2-RC1
so that expected errors are returned to the driver. #4309
System improvements
- [DocDB] Improve fault tolerance by enabling exponential backoff mechanics for the leader attempting to catch up the follower. If this causes any issues, you set the
--enable_consensus_exponential_backoff
flag (enabled by default) tofalse
. #4042 - [DocDB] Improve row scanning by using SeekForward for intents. In testing, performance of
SELECT COUNT(*)
has improved by 66%. #4277 - [DocDB] Add asynchronous transaction status resolution to conflict detection. #4058
Yugabyte Platform
- When performing a full move or add node on a universe that has a yb-master, the
server.conf
file is now being updated with the newmaster_addresses
. #4242 - In the Backups tab, individual YSQL tables can no longer be selected. Previously, attempting to back up a YSQL table would create a failed task. #3848
- In the Metrics view, transactions have been added to the YSQL and YCQL operations charts. #3827
- Create Read Replica and Edit Read Replica pages are no longer in beta. #4313
- In the Certificates page, you can now download certificates. #3985
- In the Universes overview page, add a button to toggle on metrics graph widgets to auto-refresh or to set refresh interval. #2296
Note: Prior to 2.0, YSQL was still in beta. As a result, 2.0 release includes a backward-incompatible file format change for YSQL. This means that if you have an existing cluster running releases older than 2.0 with YSQL enabled, then you will not be able to upgrade to version 2.0+. Export from old cluster and import into a new 2.0+ cluster is needed for using existing data.
v2.1.5 (April 27, 2020)
YSQL
- [BETA] The
CREATE TABLE
statement now supports theSPLIT AT VALUES
clause. #1486 - Read restart for
EXECUTE
statement if the prepared statement isSELECT
. #4205 - Find and list YSQL tables in
yb-admin
commands now use table ID instead of table names. #1687 - The
DROP INDEX
statement now ignores index backfill. - The
DROP INDEX
statement now ignores DocDBNotFound
errors when it doesn't have table metadata, but postgres does. #4249 - Force single row update prepare statements to use a custom plan that requires
boundParams
to be sent for creation and execution. #4219 - Improve performance of multi-column primary keys by generating
scanspec
for range partitioned tables using condition expression. #4033
System improvements
- Improve tablet splitting #4169, including:
- Add support for transaction-enabled tables.
- Add WAL index flush before copying WAL during table splitting.
- [colocation] Optimization that pushes index lookup down to DocDB. #3609
- [colocation] Use range keys by default for colocated tables and indexes. #3034
- [colocation] Avoid excessive RPC requests for drop and truncate #3387
- On starting a new
yb-master
(in edit and add node), update master addresses correctly inyb-master
andyb-tserver
configuration files. #3636, #4242, and #4245 - Add backup-related code changes for snapshots #3836, including:
- Change
yb-admin import_snapshot
to return an error if there are less new table names than backed up tables. For example, if you rename only two of three tables, an error will be generated. - Change
yb-admin restore_snapshot
andyb-admin list_snapshot
to outputrestoration-id
(useful for verifying completed restorations).
- Change
- Support
yb-admin import_snapshot
renaming only a few tables (but not all), but keeping the specified table name the same as the old table name. #4280 - Deprecate
table_flush_timeout
in theyb-admin create_snapshot
command. - Do not return error in output of
yb-admin get_is_load_balancer_idle
if load balancer is busy. [#3949] - Change
yb-tserver
/utilz
endpoint page to display "Live Ops" instead of "RPCs" and add YSQL statements link #4106 - Fix access to reset tablet peer during shutdown. #3989
- GetSafeTime should wait instead of adding to safe time. #3977
- Add retry logic to snapshot operations. #1032
- Add TLS encryption support to
yb-ts-cli
(adds--certs_dir_name
flag) for sending secure RPC requests to the servers. #2877 - Fix
yb-ctl
failing when passingvmodule
in--master_flags
. #4234 - The
yugabyte-client
package now includes ashare
folder (containing.sql
files) for use by Yugabyte Cloud and other remote client users. #4264
Yugabyte Platform
- When shrinking a universe, remove nodes in descending index order. #3292
- Add back up and restore of Yugabyte Platform using
yb_platform_backup.sh
script. #4208 - Change
yb_backup.py
to use theyb-admin
changes for backup-related changes for snapshot (see System improvements above). - Fix expected restoration state in the
yb_backup.py
script. - Allow users to select multiple single tables to backup in addition to specifying a full universe backup. #3680
- Customize the SMTP server for sending alert messages using configuration entries for
smtpData
(smtpServer
,smtpPort
,emailFrom
,smtpUsername
,smtpPassword
,useSSL
, anduseTLS
). #4201 - [YW] Add option to specify table keyspace when creating manual or scheduled backups. #3342
- For Azure Storage blob backups, use SAS tokens instead of Service Principal client secrets.
- Add create and restore backup support for Azure Blob Storage with SAS tokens. #3721
- [YW] Add IAM Role toggle in provider storage configuration to use the IAM role instead of requiring an Access Key and Secret. #4204
- [YW] When creating a universe and AWS provider is selected, display new Use IAM Profile toggle and ARN String text field. #4199
- Use Raft configuration as a source for master addresses in
server.conf
for master. #4089 - [YW] If a node appears as unreachable, it can be removed or released without generating errors. #4171
- [YW] Create GCP providers with any combination of host credentials and host (or shared) VPC. #4177
Note
Prior to 2.0, YSQL was still in beta. As a result, 2.0 release includes a backward incompatible file format change for YSQL. This means that if you have an existing cluster running releases older than 2.0 with YSQL enabled, then you will not be able to upgrade to version 2.0+. Export from old cluster and import into a new 2.0+ cluster is needed for using existing data.
v2.1.4 (April 15, 2020)
YSQL Changes
- [YSQL] Buffer writes to postgres system tables, reorder table creation on yb-master. #4013
- [YSQL] Single row update overwrites unmodified columns with default values to null #4127
- [YSQL] Read restarts are not done transparently in JDBC "simple" query mode. #4094
- [YSQL] Remove redundant call of YBOnPostgresBackendShutdown. #3236
- [YSQL] Allowing a deletion of a table when the DocDB returns a 'NotFound' error. #3305
- [YSQL] Use higher priority for transactions with explicit locking statements. #4136
- [YSQL] Made pg_notify a NOOP emitting warning. #3875
YCQL
- [YCQL] Backing up table should back up its indexes. #3836
System improvements
- Fix memory leak on transaction abort. #3631
- colocation: enforce colocated reloption behavior. * Implement backfilling for YCQL Unique Index. #3891
- yb-admin hangs for full timeout of 60s when no leader master is present/specified. #209
- Use QLExprResult during expression evaluation to avoid unneccesary memory allocations and copies. #3926
- Fix LB interaction with Preferred Regions. #3461
- [yb-admin] list_all_masters does not list dead hosts. #4070
- Delete a transaction aware snapshot. #1032
- Prevent read-write data race on field in DBImpl. #4025
- [docdb] Master heartbeat writes should be batch written to raft. #2236
- TServer-side tablet splitting. #1459
- Better descripton for BackfillTableJob. #4143
- Bump max threads in the RPC thread pool to 1024. #4058
- Avoid extra fsyncs of tombstoned tablets during startup. #4006
- Fixed tablet log prefix. #4188
- Remove peers when its tablet is in the FAILED state so that they can be remote bootstrapped. #3539
- Index deletion upon backfill failure. #3978
- More resilient multi node yugabyted. #2057
- Added split-tablet RPC support into yb-admin and yb-master. #1461
- Fix master bootstrap after snapshot operations. #1032
- TabletIterator Hangs during Shutdown. #4081
- Enable Move From Leader LB. #3461
- [ybase] Set default value of copy_table_batch_size to 10K. #3743
- Retain delete markers on index tables only during backfill. #4211
Platform
- Ability to attach multiple security groups to yugabyte nodes. #1598
- Remove psql from bin and fix get_clients.sh. #3928
- Set replication_factor gflag on master when creating universe. #4099
- Support for GCS backups. #3405
- Show node name in alert emails. #4102
- [YW] Add Scheduled backups to universe level Backups tab and remove from top-right dropdown. #3591
- Disable standalone backups for YCQL indexes. #3848
- [YW] Fix map markers not showing up in expanded view of node placement map in Universe overview. #4200
Note
Prior to 2.0, YSQL was still in beta. As a result, 2.0 release includes a backward incompatible file format change for YSQL. This means that if you have an existing cluster running releases older than 2.0 with YSQL enabled, then you will not be able to upgrade to version 2.0+. Export from old cluster and import into a new 2.0+ cluster is needed for using existing data.
v2.1.3 (Apr 09 2020)
YSQL Changes
- [YSQL] Parallel queries across tablets for SELECT COUNT|MIN|MAX|SUM (*|col_id) FROM. #3624
- [YSQL] Fixed bug when processing PgsqlConditionPB::operands::column_id. #3837
- [YSQL] IndexScan returns all rows when there is no match. #3896
- [YSQL] Partial support for operator families, ALTER OPERATOR and ALTER EXTENSION. #2711
- [YSQL] Handle transaction conflict status in case of parallel read operations. #3991
- [YSQL][YCQL] Wrong results when using DocDB reverse scans with paging #4004
- [YSQL] Flush buffered operations before calculating query time. #3696
YCQL
- [YCQL] Send proper error when we query system.peers_v2 table. #3862
System improvements
- Implement paging for index backfill. #3879
- Rename indices to indexs in the context of index-backfill. #3901
- Cleaner error reporting from process main. #3876
- Fix determining the oldest SST file for intent cleanup. #3917
- Fix the TS UI transactions link. #3922
- Fix transaction status resolution with old peer version. #3916
- Add nodiscard attribute to Status class. #3918
- [docdb] Deregister callback for common mem trackers from root mem tracker in destructor of vector. #3750
- First phase of transaction aware snapshot. #1032
- [cdc] Fix data races in CDC Poller. #3771
- [YSQL] Fix for PgGate shutdown race on shutdown_done flag. #3779
- Initial multi node yugabyted version. 2057
- Enable checksum verification for meta blocks of encrypted files. #3974
- Batch copy table operations that happen during create database. #3743
- yb-admin: flush_table_by_id, compact_table_by_id #3814
- [docdb][ysql] Reuse iterators during txn conflict resolution. #3521
- Persist backfilling timestamp across master failures. #3611
- Relax requirement on number of log segments in QLTransactionTest.PreserveLogs. #3989
- Fix encryption format for newly created files. #3976
- Move remote bootstrap call to OpenTablet to the threadpool. #3012
- Support the ability to fetch the oldest timestamp record that is more recent than a specified time. #4019
- A flag for veryfing meta block checksums. #4023
- Stop cleaning intents SST files if previous file was not actually deleted. #3917
- Fix MakeFuture utility. #3781
- Restore transaction aware snapshot. #1032
- Fix demo destroy behavior with Python 2.7.5. #3993
- Adding health check for the Tserver. #3540
- Don't block remote bootstrap service during session init. #4035
- YBClient sbould wait for all sync operation to complete upon shutdown. #3989
- Inherited permissions shouldn't replace granted permissions in the client's cache. #4062
- Fix race condition on ParseTree::stale_. #3083
- Handle RPC register failure in TransactionStatusResolver. #4064
- Handle tablet not running in /transactions page. #4071
- Propagate history cutoff from leader. #1032
- Allow yb-admin to work with any non-leader master. #4000
- Transaction aware snapshot persistence. #1032
- Optimize returning results from tserver to postgres. #3926
- Allow tracing recent operations in MvccManager. #4108
- Persist ALTER ROLE changes to disk. #4105
- colocation: cache db colocated in postgres #4101
Platform
- Fix edit flow to show encryption at rest flags properly. #3910
- Add support for disabling readonly users. #3794
- Add API to get client certificate. #3984
- Create GCP provider with shared VPC and allow specifying subnets per region. #3383
- Update AWS price info && stop auto-running update script.
- Remove devops
subnetId
requirement when specifying a VPC when creating GCP provider. - Add node ip to all created universes
- Omit master servers from being counted toward total node/pod count. #4009
- Update master selection logic. #3999
- Prevent data move if there is no data to move. #4046
- UI to set region-subnet mapping when creating GCP provider for existing VPCs. #3381
Note
Prior to 2.0, YSQL was still in beta. As a result, 2.0 release includes a backward incompatible file format change for YSQL. This means that if you have an existing cluster running releases older than 2.0 with YSQL enabled, then you will not be able to upgrade to version 2.0+. Export from old cluster and import into a new 2.0+ cluster is needed for using existing data.
v2.1.2 (Mar 10, 2020)
YSQL Changes
- [YSQL] Framework for expression pushdown. #3297
- [YSQL] Optimize relcache preloading by reducing the number of master RPCs. #3049
- [YSQL][Diesel ORM] Fix crash for nested SELECT with set-returning function and LIMIT. #3811
- [YSQL] Fix error when ybctid value matches the partition boundary. #3833
- [YSQL][Hasura] Fix 'Missing base table ybctid' error for UPDATE with RETURNING clause. #3805
- [YSQL] Update on table with multiple FK tables crashes postgres. #3583
System improvements
- [cdc] Fix DeleteTableWithCDCStream
- Ensure only Tablet leaders can confirm alter table. #2236
- Make OpenSSL usage thread-safe. #3454
- Revert removed timeout from RequestConsensusVote. #3390
- colocation: fix DFATAL on UI tables page (#3726). #3726
- Ignore writes from the same transaction while updating CQL indexes. #3718
- Update setting for default number of compaction threads. #3561
- Add functionality of resolving intents to transaction participant. #1032
- [cdc] Ability to Delete & List CDC Streams in yb-admin. #3638
- Set default manifest preallocation to 64k. #3666
- A workaround for encryption counter overflow. #3707
- Fixed SecureStream OOM during network partition. #3842
- Transactions could hang at transaction participant after write stopped. #3823
- Show backfill progress at master/tasks. #3668
- Fix apply of transaction with deleted tablets. #3828
- Comment out Connection::Shutdown info log. #3842
Platform
- Fix health check failures when YSQL auth is enabled. #3737
- Adding under-replicated tablet servers to the master health check. #1374
- Fix Tab problem when going into Tables page on Yugabyte Cloud. #3749
- Only save usable AWS instance types. #3626
- Added APIs for querying audit table. #3792
- Generate Client Certificate for TLS enabled Universes. #3839
- Onprem - allow specifying homeDir, automatically exclude installing s3cmd and ntp. #3834
- Add sorting by default to the releases list and also a search bar to allow users to quickly access the desired release by entering part of the string. #3563
- Correctly disable ntp install for onprem
Note
Prior to 2.0, YSQL was still in beta. As a result, 2.0 release includes a backward incompatible file format change for YSQL. This means that if you have an existing cluster running releases older than 2.0 with YSQL enabled, then you will not be able to upgrade to version 2.0+. Export from old cluster and import into a new 2.0+ cluster is needed for using existing data.