Redis Enterprise Workshop: Architecture, JSON and Search for Developers

Data Science Innovators Conference 2023Building performant applications with RedisWatch the full session · 1 hr 53 min

In a workshop at the Data Science Innovators Conference 2023, "Building performant applications with Redis," two Redis professional services leaders walked developers through how Redis Enterprise is built and how to program against it. Tom Solon, a professional services training engineer at Redis, covered architecture, data structures, sharding, JSON and search. Al Cole, vice president of professional services at Redis, then ran demonstrations of sample applications.

The workshop opened with audience polls on Redis experience, caching and search tools in use, and which follow-up workshops attendees wanted. Redis said it would send a recording and resources afterward.

How is Redis Enterprise different from open source Redis?

Solon described Enterprise as a layer on top of the open source engine. "If you think of redis Open Source as the engine," he said, "then redis Enterprise is the car built around it." Both deliver sub-millisecond latency, but Enterprise adds a proxy, a cluster manager and a REST API.

A cluster is made of nodes, databases are divided into shards, and every node runs every process, a design Solon called shared nothing. That allows multiple isolated databases on one cluster, such as separate development, test and production databases with their own users and roles.

Solon said the proxy lets clients connect to a single endpoint while it routes requests to the right shards and batches requests bound for the same shard. He also described active-passive, active read replica and active-active replication across clusters, with up to five active-active clusters.

How does Redis Enterprise handle failures?

Solon explained that quorum in Enterprise is decided by node, not by data copies, so a cluster can add a small quorum-only node to break ties instead of paying for another full-size data node. He gave an example from AWS on-demand prices at the time: a little over a dollar an hour for a large instance versus about ten cents for a small one.

If a node fails, a replica shard is promoted and a new replica is created. Persistence runs from the replica so that disk input and output does not slow the shards serving clients. With rack-zone awareness turned on, he said, losing a whole rack leads to an election and promotion of replicas in about a millisecond or so.

Which Redis data structures fit which use cases?

Redis data structures mirror those in programming languages, which Solon said reduces the "impedance" developers face translating between code and relational tables. He tied each structure to a use case:

1. Strings for caching, with expiration set in the same command.
2. Hashes for session stores, where a field such as a hit count can be incremented without rewriting the whole session.
3. Lists for queues and stacks.
4. Sets for recommendations or fraud checks, using intersections, unions and differences.
5. Sorted sets for real-time game leaderboards, and geo sets for queries such as pharmacies within 600 meters.
6. Pub/sub for service-to-service messaging. Enterprise hashes the channel name so a channel lives on one shard.

What should developers know about sharding and multi-key commands?

Solon said a single open source instance handles about 25 gigabytes and 25,000 operations per second while keeping sub-millisecond latency, and that sharding lets databases grow to tens or hundreds of terabytes. Keys are hashed with CRC16 into hash slots.

Then comes what he called a wall. "Multi-key commands only work if everything's in the same hash slot," he said. Transactions and Lua scripts have the same limit. Enterprise's proxy handles multi-key delete, multi-set and multi-get automatically. For everything else, developers can use hashtags, curly braces around part of a key, so related keys such as a user's alerts, messages and status land in one slot.

In a demo on Redis Insight, a free tool, he showed a union of two sets fail with a cross-slot error, then succeed once both set names shared a hashtag.

How do Redis JSON and Redis Search work?

Solon said the JSON module stores documents natively, so developers can read or change a single nested property, append to an array or increment a number atomically without loading the whole document. Storing JSON as a serialized string, he said, makes element access impossible and updates non-atomic.

Redis Search indexes hashes or JSON documents. The documents stay schemaless, but the index has a schema with text, numeric, tag and geo fields, plus options such as weights, sorting and no-stemming. The index updates automatically as documents change. In Enterprise, the proxy queries every shard's index and combines the results. Solon demonstrated a real estate search combining city, bedrooms, square footage, garage and distance from a park, plus an aggregation of average sale price.

Cole then used Redis App Studio, a sample application suite, to load an HR spreadsheet and a Best Buy product dataset of about 1,000 records. He showed create, edit and delete operations, a filter builder, autocomplete suggestions and faceted product search. The tool shows the Redis commands behind each action, which he called a teaching feature.

Where can you learn more about enterprise data infrastructure?

Developers weighing infrastructure choices for AI applications can hear from practitioners at COLLIDE 2027, the largest AI conference in the Southern U.S. It takes place Wednesday, October 13, 2027, at the Sandy Springs Performing Arts Center in Atlanta. Details are at /events/collide/, and registration is coming soon.

Frequently asked questions

What is the difference between Redis Enterprise and open source Redis?

Solon described Enterprise as built around the open source engine, adding a proxy, cluster manager, REST API, multi-tenancy and cross-cluster replication.

Why do multi-key commands fail in a clustered Redis database?

They only work if all keys are in the same hash slot. Hashtags in curly braces make related keys share a slot.

What does Redis Search index?

It indexes hashes and JSON documents using a schema of text, numeric, tag and geo fields, and updates the index as documents change.

What is Redis App Studio?

A sample application suite Cole used to demonstrate Redis core, JSON and search, showing the commands behind each action.

This article is based on remarks by Al Cole, Vice President, Professional Services, Redis; Tom Solon, Professional Services Training Engineer, Redis at Data Science Innovators Conference 2023 on April 12, 2023. More coverage: More AI Connect Events.

Filed under: Data & AI Infrastructure, AI Use Cases & Applications