v1.0.0 (May 01, 2018)

YugaByte DB 1.0 release 🔥🍾

Overview of changes

Big payload!

• PostgreSQL API beta support
• Cassandra API enhancements (now called YugaByte Cloud Query Language or YCQL):
  • Introduced a JSONB data type for storing documents
  • Added a personalization and a timeseries workload to the sample apps
• Redis API enhancements (now called YugaByte Dictionary Service or YEDIS):
  • Introduced follower reads
  • Added more command coverage (like ZRANGE, TSLastN, etc)
• Distributed transactions enhancements
  • Increased transactions perf
  • Fixed some corner case error conditions
• Lots of performance improvements
  • Improved core decoding logic to boost perf on 2-core machines
  • Better handling during huge spikes (backpressure)
  • Handle large network buffers better when getting large memory-buffer requests

Key Commits

• [0d07add] #239 Callback should keep a reference to Synchronizer in UpdateReplica
• [10338ff] Consensus: Use periodic timers for failure detection
• [f1e12eb] Loose-ends for PostgreSQL support
• [e9f3735] Table splits should be created based on primary cluster tserver count.
• [193cd4c] (#162) Add grammar and parsing for basic json where clause filtering.
• [0bc9d69] #198 Show only Cassandra keyspaces and tables in cqlsh
• [39ef45d] Fix bug in Postgres SSL negotiation handling
• [95aa22e] Read replicas should not show up as LEARNERSs
• [a5bbca4] Avoid using spinlocks in the ts_tablet_manager.
• [ec4ae20] Fix value validation in CassandraTransactionalKeyValue load-tester.
• [c19b14f] Fix intermittent "Pool is CLOSING" exception in the write-only setup phase of Java load-tester
• [c560765] Adding java support for shared thirdparty
• [c52c55e] Automatically set network buffer limits based on available memory
• [7ba423c] LogCache improvements: SpaceUsed outside lock and fix race on next_sequential_op_index_
• [7892a2a] Limit max wait in SetPermanentUuidForRemotePeer
• [9bf8fce] Use atomic variable for state in TabletPeer
• [43cda3b] Further reduce writers for batch CQL workloads.
• [7458489] Removed filenames from error messages
• [e0b4b5d] Add links for Cassandra, Redis and PGSQL API RPCs in progress in tserver.
• [aad076c] Reduce default logging from reactor.
• [be18db9] Fix TSAN version of TestRedisService.TestHMGetTiming
• [bb204e4] #236: Fixed TestPathUtil.TestODirectFileCreationInDir
• [26552c5] Linting error for common due to new lib yb_bfpg
• [e86a336] #197: Fix DoсDB compaction filter to skip transaction metadata
• [222341b] Switch RaftConsensus to shared_ptr
• [3cdeecf] Load balancer does not respect replication factor
• [a9fcc08] #196: Fix "Commit of expired transaction" error under high-load distributed transactions
• [9b6df54] Harden CreateTable validation in CatalogManager
• [5dc30fc] Improve DocDB and IntentAwareIterator performance
• [92ac433] Keep pending operation counter disabled on tablet shutdown
• [49c9d4a] Postgresql Support
• [7b12cce] Allow integration of distributed txns with NTP clocks such as AWS time sync service
• [d6d130b] #194: Some java client wait APIs could incorrectly exit due to overflow.
• [b125942] Fix postgres port in java mini cluster
• [7482468] #170: New TsRevRangeByTimestamp redis command
• [8797e39] Implement ZRANGE for Sorted Sets.
• [158def7] Improve performance of IntentAwareIterator
• [91e6e02] #159: Added psql 3rd party
• [9ba8436] Async DNS resolution of peers
• [a7d1447] Removed exclusive index bounds in ZRevRange.
• [d0679af] #23: Use cassandra driver's DNS resolution
• [b67e03c] #82 Do not tombstone a tablet even if new tserver doesn't get promoted
• [d537ed2] #87 Expire based on time spent in the queue
• [1796740] (#141) DocDB filtering on minor compactions
• [08762c7] #148: Fixed clqsh wrapper
• [be2d598] Basic jsonb implementation with support at C++ SQL processor level.
• [c9d74a2] #110: Allow redis reads from followers
• [d78db0f] Remove transaction_table_num_replicas flag.
• [e640ac7] Add support for arrays in our jsonb serialization format.
• [05d100c] Additional data types for jsonb serialization.
• [907243a] (#109) Support TSCard for redis time series.
• [17c3081] Make the appenders use a thread pool
• [c2555ed] Allow YBClient callback(s) to run in a separate threadpool
• [9cd6a2f] Limit the total amount of memory consumed by network read buffers
• [8f2f390] add gflag to enable/disable compression of sstables
• [29058a0] #109 Implement TSLastN functionality for redis time series.
• [89a0269] Change defaults for Kernel Watchdog
• [[4c5813a](https://github.co...

v0.9.8 (Mar 21, 2018)

Overview of changes

• Basic jsonb serialization and de-serialization to support a document datatype
• Lots of performance improvements
  • Improved performance when running on 2 core machines with auto-tuning of various gflags
  • Reduced lock contention
  • Eliminated unnecessary memory copies
• YugaByte DB can now run on ZFS filesystem
• Initial secondary index read/write support
• Handling large requests in Redis and memory protection on violating max command size

Key commits

• [cdf48e3] Fix hybrid time going backwards because of logical component overflow
• [0bf401e] Add RPC read buffers to MemTracker
• [451b7f8] Misc client template simplifications
• [e27d429] #103 Do not crash due to block size mismatch on ZFS
• [218ddcd] Remove log spew from TabletServiceImpl::Read
• [956deb2] #101 pick better auto-defaults for max number of compactions
• [28ca802] Keep java client API to pass in number of tablets for redis table creation.
• [e321311] Secondary index read/write
• [e463718] Fix macOS build issue in physical_time.cc
• [f4a1c89] Fix build in pg_server_main.cc
• [5a91590] #1 Enable building on gcc 7 on Ubuntu 17.10
• [6f9bdef] Add version info to log file including build timestamp and git
• [f9bafcc] New clock abstraction layer
• [2826097] Change defaults on server side for some flags.
• [0440305] Fix missing comma in ybctl for pgsql service display.
• [a18d393] Remove number of tablets from redis table creation client APIs.
• [26f909e] RPC support for processing commands from Postgresql clients.
• [40786a4] #97: Fix for Fatal failure on scaling down kubernetes cluster
• [a60c81f] Report the stored log entry header CRC in the CRC mismatch error message.
• [4c6baeb] Remove semi-colon requirement after "begin transaction" plus misc code cleanup.
• [5104ad2] #98: Remove pointer to TableReader from MultiLevelIndexReader
• [4327930] Quick fix of SegFault during reads.
• [44640b0] Avoid copying of QLValue and QLValuePB.
• [960f19f] Do not print a failure message when remote bootstrap is started successfully
• [5b16cf1] Send num shards parameter to master from yb-ctl.
• [c311ad3] Master removal needs to remove yb-data contents for shell mode.
• [2726eac] Resolving bug where the placement info that gets allocated to a master/tserver is off by one respect to the command line argument.
• [280fdc0] Fix ScopedPendingOperation to return Status::Busy error when unavailable.
• [d16bc28] Backoff exponentially for server too busy errors
• [e0b78dc] Set upperbound for intent iterator in IntentAwareIterator.
• [7147469] Miscellaneous test improvements, and better diagnostics before a fatal error in MvccManager::AddPending
• [6be69a8] Handle case when read buffer is full and command is not fully transferred yet
• [72de5cb] Put ValueType::kIntentPrefix's before all non-intent ValueTypes.
• [ed45995] Handle dependencies for libraries in mac_library_packager.py
• [556f710] TSAN Race Condition between Register and Load Balancer.
• [e2f4b9b] Fix the increment of SQLProcessor_ExecuteRequest metric.
• [2871820] #93 Fix trying to read ahead of the log while preparing peer request
• [b1d4d7f] #90: Reduce lock contention in TabletInvoker::IsLocalCall
• [0250c13] #88: Fix return value from XX and NX options
• [63363df] Basic jsonb serialization and de-serialization code.
• [897a925] #86: Accept lowercase options for SET
• [679b57b] Fix bug timezone parsing for timestamp literals
• [4902d06] Allow flushing all tablets of a given table

v0.9.7 (Mar 07, 2018)

Overview of changes

• Enables high data density by turning multi-level indexing on by default. This allows efficient queries even where there is a lot of data per node.
• Core performance fixes to run better on lightweight (2 vCPU) machines
• Better handling of huge io spikes. The scenario is having many more requests from the clients than what the database cluster can handle.
• Tuning the Redis query layer performance

Key commits

• [a4498b4] #88: Ensure we check TTL when we look up the type of a subdoc
• [beae94d] Remove Scoped watch stack from outbound calls. Gflags for consensus log
• [abe90b1] Make the appenders use a thread pool
• [6b33e8d] Bundle yb-prof.py in the release package.
• [e94574c] Change defaults for the number of concurrent tserver-to-master RPCs and Redis buffer size
• [e96545a] Allow std::atomic usage with clang and libstdc++
• [13724f5] Remove unnecessary encoding and decoding from DocDB
• [b36c90e] Spawn a shell master when adding a new master.
• [702d528] Use multi-level indexes by default
• [72ffc21] Fixing reactor-test: DelayedTask::AbortTask should still call the callback for tasks that have not been scheduled
• [c0c4c7b] Add paging to table iterator to fix RaftConsensusITest.TestAutoCreateReplica

v0.9.6 (Mar 01, 2018)

Overview of changes

• Support for using Kubernetes style cnames in YugaByte CQL client natively! The sample applications can easily be run as a Kubernetes service now.
• Lots of perf fixes for running YugaByte on machines with large number of CPUs.
• Added built-in time functions such as now(), totimestamp(), tounixtimestamp() and unixtimestampof().
• Support for high-performance batch transaction codepath with intra-batch dependencies.

Key commits

• [fadfd84] Use a separate thread pool for consensus service
• [71fc829] Initialize being_compacted to false
• [db55a6f] Support Kubernetes style cname in YugaByteDB clients
• [c27f929] Correct apply of repeated keys in txn
• [8ceaeb9] Enable creating multiple outstanding pipelines per IO thread
• [699a58e] Handle intra-dependency in batch request / transaction
• [39100f4] Replace upgrade lock with shared_lock and unique_lock
• [f17b2a8] Implementing now(), totimestamp(), tounixtimestamp() and unixtimestampof() functions for YQL.
• [024730e] Use circular buffer for receive
• [c619be7] Reserve the number of subkeys from ParseCollection
• [de6a268] Handle intra-dependency in batch request / transaction
• [a26f8d2] Need to remove the special tserver service queue sizes on "low-cpu" machines.
• [3857155] Change distributed transaction test cases to test writing to the same hash / primary key in the same transaction.
• [99d4adb] Group lookup tablet requests
• [0bd8878] Set iterate_upper_bound to avoid seeking over deleted keys in RocksDB.
• [88cf84e] Introduce limits for queued commands and don't drop connection on read buffer overflow
• [2250acb] #59: Set GIT_DIR to /tmp when building AWS SDK so that it does not pick up YugaByte DB version as its own
• [b4a9b8e] Reduce log spew for unavailable peer
• [0261099] Make Redis read buffer size and number of concurrent master lookups in meta cache configurable
• [f3e9672] Gflag for block cache num shards
• [b0553d9] Porting nonredundant functionalities of local_cluster_ctl to yb-ctl.
• [dc6e2e2] Log spew during tablet startup under load.
• [4ad88df] system_redis.redis should not be counted as an user table
• [7302b1f] Correct handling of failure to retry RPC command
• [f90a83f] Fix TransactionManager destruction

0.9.5 (Feb 16, 2018)

Overview of changes

• Support for natively running YugaByte DB using Kubernetes StatefulSets! YugaByte DB now works in GKE as well.
• Distributed transactions supported at isolation level SNAPSHOT ISOLATION.
• HINCRBY support in Redis API.
• Defaults for CQL client driver switched to strong consistency.
• Lots of performance enhancements and fixes. Notably - lock free hybric clock implementation. New YugaByte perf report uploaded.
• Improved debuggability - added slow query logging for Redis.
• Overhauled the third party build system to make it more readable and easy to maintain.

Thanks

• @mchang31415 for reporting a number of perf issues.
• @praneeth-nimmagadda for reporting issues with the COUNTER datatype.
• @nickhristov for pointing out that duplicate primary keys in a batch are not getting loaded

Key commits

• [2150bee] Added AWS C++ SDK 3rd party.
• [8d8030f] Fix TransactionManager destruction
• [9546e7e] Fix counter column datatype in system_schema.columns.
• [3074f56] Allow multiple writes of the same row within a batch request.
• [8ee1288] Update 3rd party version for mac
• [cfd54dd] Periodic timers
• [2292067] Implement HIncrBy
• [7a7421f] Fix a CHECK failure due to handling of slow responses when looking for leader master
• [bed88c9] Create cloud-machine-types-analysis.md
• [90ae07b] Update yb-perf-0.9.5rc-Feb-13.md
• [ab9deaa] Set defaults for read_pool_* flags
• [eb32648] #51: Kubernetes cname only tries the first ip address on GKE (#53)
• [2dbd7e8] Update yugabyte-statefulset.yaml
• [0c87387] Fix master UI to show only running tables
• [c294304] Capture by reference to avoid copies during HMGet
• [c6d2635] High CPU usage in CleanWaitingConnections
• [d4fd22c] Print slow queries for Redis. Similar to CQL
• [6488bfd] Set GLOG flag so that we do not sync for WARNING messages
• [e230091] Fix parsing of tcmalloc max thread cache bytes.
• [54121c0] Added tests for ImportSnapshotMeta.
• [072595a] Update Cassandra Java driver versions to use default YB_STRONG consistency level.
• [c62d02a] Make HybridClock Now/Update lock-free
• [4fa3118] Parallelize reads for Redis ops in the same TabletService::Read request
• [8cd931c] Increase tcmalloc local thread cache size for tservers.
• [3ef0019] Reuse iterator for Redis Operations
• [dfb16c9] Prevent core dumps for LOG(FATAL) and CHECK.
• [07ffec9] Fix QLTabletTest.SkewedClocks and other safe time logic in MvccManager
• [58c97d3] Change YQL transaction isolation level to SNAPSHOT_ISOLATON.