Redis Playground

An observable dev environment for understanding Redis

This is a .Net Core 3.1 app using the StackExchange.Redis client library to communicate with redis and instrumented with OpenTelemetry + Jaeger for traces and Prometheus + Grafana for metrics all running in docker-compose.

Getting Started

• Run ./start.sh to run the docker-compose and start the app.
• Redis.Playground.App
  • http://localhost:5000/Queue/Push/{queueName}/{messageCount}
  • http://localhost:5000/Queue/Size/{queueName}
• Grafana - http://localhost:3000 (admin/admin)
• Jaeger UI - http://localhost:16686
• Prometheus - http://localhost:9090
• Grafana Dashboards
  • Redis Dashboard for Prometheus Redis Exporter 1.x
  • prometheus-net - AspNetCore summary
  • prometheus-net.DotNetRuntime CLR summary
  • ASP.NET Core - controller summary
  • Docker - docker-compose containers summary

Reading Materials on Redis

• Understanding The Top 5 Redis Performance Metrics - DataDog
• Redis Essentials - Packt

TL;DR Redis High Availability

• Redis Sentinel
  • The goal of Redis Sentinel is to provide reliable automatic failover in a master/slave topology without sharding data
  • Sentinel is a separate process from the Redis server and it listens on its own port
  • sentinel.conf stores the location of the master node
    • this is updated/re-written on failover, or when new sentinel or slave joins
  • Communication between sentinels happens via a pub/sub channel on the master
  • Sentinels PING master, and if a quorom of sentinels don't receive PONG from the master after "down-after-milliseconds" amount of time then they trigger failover
  • CAP theorem analysis suggests Redis Sentinel is not strongly consistent in the case of network partition
    • https://aphyr.com/posts/287-asynchronous-replication-with-failover
    • http://antirez.com/news/56
  • Requires a client that supports Redis Sentinel
  • Developed and released before Redis Cluster when Antirez had less experience in distributed systems
• Redis Cluster
  • The goal of Redis Cluster is to distribute data across different Redis instances and perform automatic failover if any problem happens to any master instance
  • Single process, but requires 2 ports (one for Redis server process, one for communicating between Redis instances)
  • Requires AT LEAST 3 masters to be considered healthy.
    • You should have at least one replica per master, otherwise if a master fails its data will be lost
    • Furthermore the entire cluster may become unavailable under the default configuration if no replicas are present
      • cluster-require-full-coverage is the configuration that controls this behavior.
        • it defaults to yes meaning the entire cluster becomes unavailable if some hash slots are not reachable
        • setting it to no means that queries routed to hash slots that have become unreachable simply return an error while queries routed to reachable hash slots remain available
    • In addition multiple replicas per master are recommended as that way when a master fails and one of its replicas is promoted to master then it still has a replica, otherwise it wouldn't have a replica and if that master were to fail it could make the cluster unavailable
      • to save on cost it is possible to configure most masters with only a single replica and one master with two replicas and provided cluster-migration-barrier is set to 1 when a master with only a single replica failed and failover was triggered then afterwards the "spare" replica from the master with multiple replicas would switch to become a replica of the newly promoted master
    • Promoting a slave to a master takes some time and data on the failed master will be unavailable for a short time until the failover is complete
  • All data is sharded across masters and replicated to slaves.
    • data is partitioned across 16,384 "hash slots" with each master owning a portion of the slots
    • var hashSlot = crc16(redisKey) % 16384
    • there is no automatic redistribution of slots across masters
      • by default they are distributed evenly across the masters when you create the cluster unless you specify otherwise
    • any multi-key operations require all the keys to be stored on the same node
      • "hash tags" can be used for this
        • The hash tag is delimited by curly brace and ensures the same hash slot is chosen for all keys
      • SADD {user123}:friends:usa "john"
      • SADD {user123}:friends:brazil "bruno"
      • SUNION {user123}:all_friends {user123}:friends:usa {user123}:friends:brazil
  • Requires client to support Redis Cluster
    • redis-cli requires passing -c to enable cluster mode, else it will treat Redis as a single instance
    • CAP Theorem analysis
      • also still not strongly consistent under network partition scenarios
• RedisRaft
  • This is a Redis Module designed to provide a strongly consistent clustered Redis deployment
  • https://github.com/RedisLabs/redisraft

150+ Talks on Redis

• Use Cases

  • TL;DR

    • Redis: Swiss army knife

  • Caching

    • Techniques to Improve Cache Speed
    • Leveraging Redis 6 Tracking for Awesome Client-Side Caching - RedisConf 2020
    • Techniques for Synchronizing In-Memory Caches with Redis
    • Increase Application Performance with SQL Auto Caching; No Code Changes - RedisConf 2018
    • Why Your MySQL Needs Redis
    • Declarative Caching with Postgres and Redis
    • Improve Cache Speed at Scale - RedisConf 2020
    • Scaling applications with Azure Redis Cache and Machine Learning
    • Scaling SQL Write-Master Database Clusters
    • Quickly scaling database clusters with Redis
    • Write-behind and read-through with RedisGears

  • Session Management

    • Creating a Highly Available Persistent Session Management Service - RedisConf 2020
    • Session Management - Redis Labs
    • The Life of a User Session - Redis Day 2019

  • Rate Limiting

    • Rate-Limiting 30M Requests a Day - Redis Day Bengaluru 2020
    • Protecting your API with Redis
    • Rate Limiting with Redis
    • Rate limiting/monitoring with Redis - 2014
    • Rate limiting APIs with Redis + Express.js
    • Throttling requests before they hit your application
    • High Volume Ratelimiting with Redis
    • API Rate Limits with Spring Boot and Redis
    • How to Protect Your APIs with Rate Limiting and Metering
    • Approaches for application request throttling - DotNext

  • Queues / Messaging

    • Redis as a Message Bus
    • Build a Message Bus with Redis Streams and FastAPI
    • Building Scalable, Distributed Job Queues with Redis - RakuConf 2012
    • Delayed Message Processing with Redis Streams
    • Distributed Queue using Redis - DSConf 2020
    • Fail Safe Starvation Free Durable Priority Queues in Redis
    • Resilient Testing At Scale Using Redis Queues to Manage Concurrency
    • Redis Pub/Sub for Fast Data Ingest
    • High-performance messaging with Go and Redis
    • Cancellable Timeout Queues, A Use Pattern
    • Increasing Message Delivery Rates with Redis

  • Analytics / AI / IoT

    • Redis Analytics Use Cases
    • Real Time Log Analytics Using Probabilistic Data Structures in Redis
    • How to Use Bloom Filters in Redis
    • Inline Analytics Redis Fraud Mitigation
    • Efficient analytics with Redis Bitmaps
    • Redis as a Time Series Data Store
    • Making Real Time Predictive Decisions with Redis
    • Real-Time Redis Deep-Learned Detection at the Edge - RedisConf 2020
    • Using Redis for Schema Detection
    • Identifying Fraudulent Online Behavior
    • High-performance real-time recommendations
    • Redis as a scalable feature store

  • Other

    • Building Highly Concurrent, Low Latency Gaming System - Redis Day London 2019
    • Using Redis and RediSearch Module to Store & Search Volatile Data - RedisConf 2017
    • Unorthodox Redis Use Cases
    • Build, deploy, and run real-time applications with Redis - RedisConf 2021
    • Redis as a primary database

  • Microservices / Application Architecture

    • Implementation patterns to leverage Redis to turbo charge existing Legacy applications
    • Microservices and Redis: A Match made in Heaven
    • Where is my cache? Architectural patterns for caching microservices by example
    • Building Lightweight Microservices Using Redis
    • Event Driven Microservices with Event Sourcing and Redis
    • Redis Streams for Event-Driven Microservices
    • Building Lightweight Microservices with Redis and Hydra
    • Using Redis Streams to Build Event Driven Microservices and User Interface
    • Using Redis Streams to Build More Resilient Services - Redis Day London 2019
    • Microservices and Redis - RedisConf 2017
    • Interview: Chris Richardson Of Microservices.io
    • Common Redis Use Cases for Cloud-Native Apps and Microservices
    • Microservices Architecture in the Real World
    • Microservices w/Redis on Kubernetes + ElastiCache - RedisConf 2021

• Redis Concepts / Internals

  • Data Structures

    • Redis Data Structures for Non-Redis Users
    • Intro to Redis Commands and Data Structures
    • Intro to Redis Data Structures and Pub/Sub vs Redis Streams
    • Redis Hashes Explained
    • Redis Lists Explained
    • Redis Sets Explained
    • Redis Sorted Sets Explained
    • Redis Bitmaps Explained
    • Redis HyperLogLog Explained
    • Implementing a New Data Structure for Redis

  • Modules

    • Redis Streams

      • Redis Streams - Redis Day TLV 2018

    • RediSearch

      • Integrating Redis with ElasticSearch
      • Amazing User Experiences with Redis and RediSearch - RedisConf 2018
      • Add Real-Time Full-Text Search to Your Web Site with RediSearch
      • RediSearch Aggregation Capabilities
      • Getting Started With GeoCoding With Redis
      • RediSearch: Build modern apps with high speed performance
      • How we built real-time full-text website search with RediSearch

    • Redis Gears

      • RedisGears GA

    • RedisAI

      • RedisAI in Production
      • Hacking your way into machine learning with RedisAI

    • RedisJSON

      • JSON in Redis: When to use RedisJSON - Redis Day Seattle 2020
      • ReJSON = {"id": "old dog", "activity": "new trick"}
      • RedisJSON + RediSearch as a real-time document database

    • RedisGraph

      • RedisGraph 2.2: The Fastest Way to Query Your Highly Connected Data in Redis
      • A Practical Introduction to RedisGraph
      • Deep Dive into RedisGraph - RedisConf 2019
      • Redis Graph - Redis Day TLV 2018
      • RedisGraph and it's use cases
      • Building a knowledge graph using RedisGraph
      • Exploring code bases with RedisGraph
      • Building the fastest native graph database: RedisGraph

    • RedisTimeSeries

      • Intro to RedisTimeSeries
      • Time Series Data in Real Time with Redis Time Series Module - RedisConf 2018
      • Leveraging Redis Enterprise with RedisTimeSeries
      • Real-time querying and aggregating of time series data with RedisTimeSeries

    • Extending Redis

      • Redis Lua Scripts
      • Writing Redis Modules in Rust
      • Extend Redis With Modules
      • Josiah Carlson on Redis

  • Concepts / History

    • Architecture of Redis
    • The Evolution of Redis Key Expiration Algorithms
    • Anatomy of a Redis Command
    • Designing a Redis Client for Humans
    • History of Redis Replication and Future Prospects
    • The Design of Redis Streams - Antirez
    • Lessons Learned Running The Redis Open Source Project For 8 Years - Antirez
    • Redis: Client side caching "broadcasting mode"
    • A Brief History of Encrypting Your Redis Traffic
    • What Happens When Redis Runs Out of Memory

  • Optimization

    • Redis Memory Optimization
    • Redis Memory Optimization - RedisConf 2018

  • Distributed Caching System Design Concepts

    • Redis system design | Distributed cache System design
    • Caching Architectures | Microservices Caching Patterns | System Design Primer
    • What is Distributed Caching? Explained with Redis!

• Redis In Production

  • War Stories

    • Anti-patterns with Redis
    • Learn Redis the hard way... in production - DEVit 2017
    • Redis: Going Beyond Simple Failover - NYC Redis Meetup 2014
    • Processing 60 million messages a day using Redis: Lessons from the trenches
    • Troubleshooting Redis
    • Redis Networking Nerd Down
    • Redis is Dead Long live Redis!
    • Evolving Your Distributed Cache In A Continuous Delivery World
    • How Redis augments site reliability engineering

  • High Availability / Scaling

    • Build and Deploy Containerized Redis Cache for High Availability and Performance
    • Scaling Redis To 1M Ops/Sec
    • Meeting Service Level Objectives at Scale with HAProxy and Redis
    • Scaling Your Redis Workloads with Redis Cluster - AWS Online Tech Talks
    • App Sharded Redis w/Sentinel vs Redis Cluster - RedisConf 2019
    • Multi Master Redis: A Deep Dive - RedisConf 2017
    • Deep Dive into Redis Replication - RedisConf 2019

  • Case Studies

    • Airbnb's Journey from Self-Managed Redis to ElastiCache for Redis - AWS re:Invent 2018
    • Redis Cluster at Flickr Yahoo! - RedisConf 2017
    • Redis at LINE, 25 Billion Messages Per Day - RedisConf 2018
    • Monitoring and Scaling Redis at DataDog
    • Preventing Cache Stampede with Redis and Xfetch
    • How Shopify is Scaling Up Its Redis Message Queues
    • Operationalizing Redis at Scale at Square
    • Redis at Lyft: 1,000 Instances
    • Beyond Sharding @Lyft - RedisConf 2020
    • Using Redis at Scale at Twitter - RedisConf 2017
    • Redis in a Multi Tenant Environment
    • How Roblox Keeps Millions of Users up to Date with Redis Pub/Sub - RedisConf 2017
    • How GAP scaled over 100x using Redis
    • Processing 160 million messages a day using Redis with Wavy (Portuguese w/ English slides)
    • Uber's journey to Redis Cluster
    • How Adobe Leverages Redis to Serve and Secure Billions of API Requests

  • Redis Enterprise

    • Continuous Availability, Infinite Scaling, Secure, Cost-effective Redis - RedisConf 2017
    • Ultra Scaling with Redis Enterprise
    • Redis Enterprise Overview - Redis Day TLV 2016
    • Troubleshooting Redis Performance Insights w/ Redis Enterprise Cluster
    • Redis on Flash - A Game-Changing Database Platform for Etermax
    • Redis Enterprise Flash: Scaling Data Size with Redis Enterprise at Near RAM Latency

  • AWS ElastiCache for Redis

    • Extreme Performance at Cloud Scale - RedisConf 2019
    • Building Resilient Real-Time Applications with Amazon ElastiCache for Redis - AWS Online Tech Talks
    • Fast Data at Internet Scale with Amazon ElastiCache for Redis
    • Enterprise-Class Security, High-Availability, & Scalability with Amazon ElastiCache
    • Redis & Amazon ElastiCache: Building Real Time Apps - AWS 2019
    • Amazon ElastiCache for Redis Global Datastore - Active-Passive Multi-Region Redis
    • Scaling Redis Workloads with Amazon ElastiCache
    • Best practices: migrating Redis clusters from Amazon EC2 to ElastiCache

  • Chaos Engineering

    • Redis Fault Injection
    • Ensure Redis Reliability with Gremlin, Chaos Engineering

  • Performance

    • The Top Redis Metrics that Matter - DataDog 2015
    • Benchmarking Redis
    • Redis benchmark in native Linux mode, Linux VM, and Docker container
    • Redis Reliability, Performance & Innovation
    • High Performance Redis
    • Benchmarking Redis By Itself and Versus Other NoSQL Databases
    • Redis on NVMe SSD

  • Security

    • Exploiting Unauthenticated Redis - TryHackMe!
    • New Security Features in Redis 6 Open Source
    • Practical Use Cases for Access Control Lists in Redis 6
    • Secure Redis Cluster at Box
    • Hack The Box: Postman Walkthrough [Redis, SSH, Webmin Exploit]

  • Monitoring / Observability / Dev Tooling

    • RedisInsight - RedisConf 2020
    • RedisInsight - A FREE Robust GUI Tool to Monitor Redis Data
    • Improve your Redis developer experience with RedisInsight - RedisConf 2021
    • Monitoring Redis data using RedisInsight - OSConf 2020
    • Prometheus Exporter and Grafana with Redis
    • Getting visibility into complex Redis Pub/Sub transactions using OpenTelemetry