Presentation is loading. Please wait.

Presentation is loading. Please wait.

Open source, advanced key-value store, data structure server REmote DIctionary Server.

Similar presentations

Presentation on theme: "Open source, advanced key-value store, data structure server REmote DIctionary Server."— Presentation transcript:

1 open source, advanced key-value store, data structure server REmote DIctionary Server.

2 Hello Redis Install $ wget $ tar xzf redis-2.4.8.tar.gz $ cd redis-2.4.8 $ make Start Redis server & make connection [binzhang@phxrueidb04 redis-2.4.8]$ src/redis-server redis.conf [16849] 04 Mar 02:03:59 * Server started, Redis version 2.4.8 [16849] 04 Mar 02:03:59 * The server is now ready to accept connections on port 6379 Connect server $ src/redis-cli redis > set player:666:name binzhang OK redis > get player:666:name "binzhang" 2

3 Learn More : data structure server 3

4 Agenda Redis Manifesto Data structures : strings, hashes, lists, sets and sorted sets. Leveraging Redis Redis Admin and maintenance The architecture of REDIS Cases 4

5 Redis Manifesto 1.Redis is a DSL (Domain Specific Language) that manipulates abstract data types and implemented as a TCP daemon. keys are binary-safe strings and values are different kinds of abstract data types. 2.Redis has persistence option but Memory storage is #1. 3.The Redis API is a direct consequence of fundamental data structures. 4.Code is like a poem. 5.We believe designing systems is a fight against complexity. Most of the time the best way to fight complexity is by not creating it at all. 6.Redis API has two levels: 1) a subset of the API fits naturally into a distributed version of Redis and 2) a more complex API that supports multi-key operations. 7.We optimize for joy. When there is no longer joy in writing code, the best thing to do is stop. 5

6 Redis Manifesto : where is Redis? 6 1.Redis is extremely fast, making it perfectly suited for applications that are write- heavy, data that changes often, and data that naturally fits one of Rediss data structures (for instance, analytics data). 2.A scenario where you probably shouldnt use Redis is if you have a very large dataset of which only a small part is hot (accessed often) or a case where your dataset doesnt fit in memory.

7 Who is using Redis 7 Taobao Sina weibo

8 Agenda Redis Manifesto Simple/fast/ Data structure : strings, hashes, lists, sets and sorted sets. Leveraging Redis The architecture of REDIS Redis Admin and maintenance Cases 8

9 Data structure: Key-Value Data Store Keys are strings which identify pieces of data (values) Values are arbitrary byte arrays that Redis doesn't care about Redis is implemented as five specialized data structures Strings, hash, list, set, sort set Pub/Sub Querying with Redis ; the above make Redis fast and easy to use, but not suitable for every scenario 9

10 Data structure : KEY value Before we dive into the specific data types, it is important to look at a few things you should keep in mind when designing the key structure that holds your data. 1.a key can contain any characters, you can use separators to define a namespace with a semantic value for your business. An example might be using cache:project:319:tasks, where the colon acts as a namespace separator. 2.When defining your keys, try to limit them to a reasonable size. Retrieving a key from storage requires comparison operations, so keeping keys as small as possible is a good idea. Additionally, smaller keys are more effective in terms of memory usage. 3.Even though keys shouldnt be exceptionally large, there are no big performance improvements for extremely small keys. This means you should design your keys in such a way that combines readability ( to help you) and regular key sizes (to help Redis). 10

11 Data structure : KEY value Redis support different kind of values 1.Binary-safe strings. 2.Lists of binary-safe strings. 3.Hash map of strings 4.Sets of binary-safe strings, that are collection of unique unsorted elements. 5.Sorted sets, similar to Sets but where every element is associated to a floating number score. The elements are taken sorted by score. 6.pubsub channels are a new addition to Redis 11

12 Data structure : strings, hashes, lists, sets, sorted sets. 1.Strings: the simplest and most basic data type (max 512M in length) 2.Redis Strings are binary safe, this means that a Redis string can contain any kind of data, for instance a JPEG image or a serialized Ruby object. – redis > users:leto "{name: leto, planet: dune, likes: [spice]}" – OK – redis > x users:leto "{name: leto, planet: dune, likes: [spice]}" – (integer) 0 redis > strlen users:leto (integer) 42 redis > getrange users:leto 27 40 "likes: [spice]" redis > append users:leto " OVER 9000!!" (integer) 54 redis > get users:leto "{name: leto, planet: dune, likes: [spice]} OVER 9000!!" 12 The C structure sdshdr declared in sds.h represents a Redis string: struct sdshdr { long len; long free; char buf[]; };

13 Data structure : strings, hashes, lists, sets, sorted sets. if we store a counter key, we can use commands such as INCR (or INCRBY) and DECR (or DECRBY) to increment or decrement its contained value. – To store page visit data, we could have a key visits:pageid:totals redis > SET visits:2:totals 1367894 OK redis > get visits:2:totals "1367894" redis > INCR visits:635:totals (integer) 1 redis > INCR visits:2:totals (integer) 1367895 redis > get visits:2:totals "1367895 redis > strlen visits:2:totals (integer) 7 redis > incr users:leto (error) ERR value is not an integer or out of range 13

14 Data structure : strings, hashes, lists, sets, sorted sets. Much like traditional hashtables, hashes in Redis store several fields and their values inside a specific key. so they are the perfect data type to represent objects (eg: A User with a number of fields like name, surname, age, and so forth): Example: Designing a key namespace to store our users. redis> hset users:jdoe name "John Doe" (integer) 1 redis> hmset users:jdoe email phone "+1555313940" OK redis> hincrby users:jdoe visits 1 (integer) 1 redis > hget users:jdoe phone "+1555313940" A hash with a few fields (where few means up to one hundred or so) is stored in a way that takes very little space, so you can store millions of objects in a small Redis instance. Every hash can store up to 2 32 - 1 field-value pairs (more than 4 billion). 14

15 Data structure : strings, hashes, lists, sets, sorted sets. HDEL key field [field...] Delete one or more hash fields HGETALL key Get all the fields and values in a hash HINCRBY key field increment Increment the integer value of a hash field by the given number HKEYS key Get all the fields in a hash HLEN key Get the number of fields in a hash HMGET key field [field...] Get the values of all the given hash fields HMSET key field value [field value...] Set multiple hash fields to multiple values HSETNX key field value Set the value of a hash field, only if the field does not exist HVALS key Get all the values in a hash redis > hgetall users:jdoe 1) "name" 2) "John Doe" 3) "email" 4) "" 5) "phone" 6) "+1555313940" 7) "visits" 8) "1" 15

16 Data structure : strings, hashes, lists, sets, sorted sets. Redis Lists are simply lists of strings, sorted by insertion order. It is possible to add elements to a Redis List pushing new elements on the head (on the left) or on the tail (on the right) of the list. LPUSH mylist a # now the list is "a" LPUSH mylist b # now the list is "b","a" RPUSH mylist c # now the list is "b","a","c" (RPUSH was used this time) support for constant time insertion and deletion of elements near the head and tail, even with many millions of inserted items. Accessing elements is very fast near the extremes of the list but is slow if you try accessing the middle of a very big list, as it is an O(N) operation. You might want to use lists in order to implement structures such as queues The max length of a list is 2 32 - 1 elements (4294967295, more than 4 billion of elements per list). 16

17 Data structure : strings, hashes, lists, sets, sorted sets. Model a timeline in a social network, using LPUSH in order to add new elements in the user time line, and using LRANGE in order to retrieve a few of recently inserted items.LPUSHLRANGE – Lrange to do paging You can use LPUSH together with LTRIM (O(N))to create a list that never exceeds a given number of elements, but just remembers the latest N elements.LPUSHLTRIM – Capped Collections in MongoDB. Capped Collections Lists can be used as a message passing primitive, – BLPOP key [key...] timeout Remove and get the first element in a list, or block until one is available – BRPOP key [key...] timeout Remove and get the last element in a list, or block until one is available – LINDEX key index Get an element from a list by its index – LLEN key Get the length of a list – LPOP key Remove and get the first element in a list – RPOP key Remove and get the last element in a list – LRANGE key start stop Get a range of elements from a list – LTRIM key start stop Trim a list to the specified range 17

18 Data structure : strings, hashes, lists, sets, sorted sets. Sets are an unordered collection of Strings. It is possible to add, remove, and test for existence of members in O(1) (constant time regardless of the number of elements contained inside the Set). Elements in a given set can have no duplicates. this means that adding a member does not require a check if exists then add operation. Sets are a natural fit for circles, because sets represent collections of data, and have native functionality to do interesting things like intersections and unions. The max number of members in a set is 2 32 - 1 (4294967295, more than 4 billion of members per set). – You can track unique things using Redis Sets. Want to know all the unique IP addresses visiting a given blog post? Simply use SADD every time you process a page view.SADD – Redis Sets are good to represent relations. – You can use Sets to extract elements at random using the SPOP or SRANDMEMBER commands.SPOPSRANDMEMBER 18

19 Data structure : strings, hashes, lists, sets, sorted sets. We want to store several circles for each of our users, so it makes sense for our key to include a bit about the user and a bit about the actual circle. (circle:jdoe:family etc) 19 redis> sadd circle:jdoe:family users:anna redis> sadd circle:jdoe:family users:richard redis> sadd circle:jdoe:family users:mike (integer) 1 redis> sadd circle:jdoe:soccer users:mike redis> sadd circle:jdoe:soccer users:adam redis> sadd circle:jdoe:soccer users:toby redis> sadd circle:jdoe:soccer users:apollo (integer) 1 redis> smembers circle:jdoe:family 1) "users:richard" 2) "users:mike" 3) "users:anna" redis> hgetall users:mike (...) redis> sinter circle:jdoe:family circle:jdoe:soccer 1) "users:mike" redis> sunion circle:jdoe:family circle:jdoe:soccer 1) "users:anna" 2) "users:mike" 3) "users:apollo" 4) "users:adam" 5) "users:richard" 6) "users:toby"

20 Data structure : strings, hashes, lists, sets, sorted sets. SADD key member [member...] Add one or more members to a set SCARD key Get the number of members in a set SDIFF key [key...] Subtract multiple sets SDIFFSTORE destination key [key...] Subtract multiple sets and store the resulting set in a key SINTER key [key...] Intersect multiple sets SISMEMBER key member Determine if a given value is a member of a set SMEMBERS key Get all the members in a set SMOVE source destination member Move a member from one set to another SPOP key Remove and return a random member from a set SRANDMEMBER key Get a random member from a set SREM key member [member...] Remove one or more members from a set SUNION key [key...] Add multiple sets SUNIONSTORE destination key [key...] Add multiple sets and store the resulting set in a key 20

21 Data structure : strings, hashes, lists, sets, sorted sets. Use set to implement tags – sadd news:1000:tags 1 – (integer) 1 – sadd news:1000:tags 2 – (integer) 1 – sadd news:1000:tags 5 – (integer) 1 – sadd news:1000:tags 77 – (integer) 1 – sadd tag:1:objects 1000 – (integer) 1 – sadd tag:2:objects 1000 – (integer) 1 – sadd tag:5:objects 1000 – (integer) 1 – sadd tag:77:objects 1000 – (integer) 1 21 To get all the tags for a given object : redis>smembers news:1000:tags 1. 5 2. 1 3. 77 4. 2 we may want the list of all the objects having as tags 1, 2, 10, and 27 at the same time Sinter tag:1:objects tag:2:objects tag:10:objects tag:27:objects

22 Set: Wildcard autocomplete Split every username to three letter chunks – Simonw => sim, imo, mon, onw Create a set for each chunk – Sim => { simonw, asimov, fasim} If the user types simo, return the intersection of the sim and imo. 22

23 Data structure : strings, hashes, lists, sets, sorted sets. 1.Every member of a Sorted Set is associated with score, that is to sort set, from the smallest to the greatest score. members are unique, scores may be repeated. 2.With sorted sets you can add, remove, or update elements in a very fast way (in a time proportional to the logarithm of the number of elements, O(log(N))). 3.Get ranges by score or by rank (position) in a very fast way. – ZADD can be used both to add items to the set and to update the score of an existing member. –The ZRANGE family of commands return items by their index position within the ordered set. The optional WITHSCORES argument returns the score for each item in the same response. –ZRANGEBYSCORE query the ordered set by score, instead of by index. zadd friends:leto 100 ghanima 95 paul 95 chani 75 jessica 1 vladimir redis > zrange friends:leto 0 -1 withscores 1) "vladimir" 2) "1" 3) "jessica" 4) "75" 5) "chani" 6) "95" 7) "paul" 8) "95" 9) "ghanima" 10) "100" 23

24 Data structure : strings, hashes, lists, sets, sorted sets. Zset as index : any time you need to look up data based on range queries, you should be storing it in a sorted set. They're indexes that you have to maintain yourself. redis > zadd hackers 1940 "Alan Kay" 1953 "Richard Stallman" 1969 "Linus Torvalds" 1912 "Alan Turing" redis > zrange hackers 0 -1 1) "Alan Turing" 2) "Alan Kay" 3) "Richard Stallman" 4) "Linus Torvalds redis > zrangebyscore hackers 1950 1990 1) "Richard Stallman" 2) "Linus Torvalds redis > zrangebyscore hackers -inf 1950 1) "Alan Turing" 2) "Alan Kay" redis > zremrangebyscore hackers 1940 1960 (integer) 2 redis > zrange hackers 0 10 1) "Alan Turing" 2) "Linus Torvalds" 24

25 Sort set : Prefix autocomplete Type binz, return binzhang 1.Turn the first 4 or 5 characters of the strings into an integer (you can imagine every char as a digit of a radix 256 number for instance, but there are better representation) and add all your usernames into a sorted set with score=integer. 2.Then using ZRANGEBYSCORE you can get all the elements between a given range. --ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count] 3.This method is much more scalable as it's an O(log(N)) thing. (zrangebyscore is O(log(N)+M) ) 25

26 sorted sets : Inverted-Index Text Search with Redis inverted index - a bunch of sets mapping terms to document IDs. 2.Assign each document an ID 3.Apply stemming and stopwords first 4.Create an inverted index, with one set per word 5.Create a set of docIDs for each term – ZINTERSTORE destination-zset number-of-zsets-to-intersect zset1 [zset2...] [WEIGHTS weight1 [weight2...]] [AGGREGATE SUM | MIN | MAX] 26

27 Learn More Data structure : Pub/Sub Redis has native support for the publish/subscribe (or pub/sub) pattern 1.receivers subscribe to messages that match a specific pattern (for instance, messages that are sent to a specific channel), procedurer/emitter to send messages to me 3.emitter and receivers to be loosely coupled-- hey dont need to know each other. The pub/sub command 1.PSUBSCRIBE pattern [pattern...] Listen for messages published to channels matching the given patterns 2.PUBLISH channel message Post a message to a channel 3.PUNSUBSCRIBE [pattern [pattern...]] Stop listening for messages posted to channels matching the given patterns 4.SUBSCRIBE channel [channel...] Listen for messages published to the given channels 5.UNSUBSCRIBE [channel [channel...]] Stop listening for messages posted to the given channels 27

28 Learn More Data structure : Pub/Sub redis > subscribe irc:football Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "irc:football" 3) (integer) 1 1) "message" 2) "irc:football" 3) "have a good day" 1) "message" 2) "irc:football" 3) "water" 28 redis > PUBLISH irc:football "Rock you" (integer) 0 redis > PUBLISH irc:football "have a good day" (integer) 1 redis > PUBLISH irc:football "water"

29 Agenda Redis Manifesto Data structure : strings, hashes, lists, sets and sorted sets. Leveraging Redis The architecture of REDIS Redis Admin and maintenance Cases 29

30 Leveraging Redis 1.Operations on KEYS 2.Big O Notation 3.Sort 4.EXPIRE 5.Transaction 6.Optimistic locking using check-and-set ( select for update) 7.Pipelining ( commands in batch) 30

31 Keys operation KEYS pattern Lists all the keys in the current database that match the given pattern. [slow] TYPE key-name Tells the type of the key. Possible types are: string, list, hash, set, zset, and none. MONITOR Outputs the commands received by the Redis server in real time. [debug purpose only] KEYS h*llo KEYS h?llo KEYS h[ae]llo. redis > type circle:jdoe:soccer Set DEL/EXISTS/EXPIRE/TTL/PERSIST/RANDOMKEY/REN AME Redis can handle up to 2^32 keys 31

32 Big O Notation How fast a command is based on the number of items we are dealing with. O(1): fastest, Whether we are dealing with 5 items or 5 million, you'll get the same performance. Sismember : if a value belongs to a set O(N) : linear commands, fts etc Keys ltrim, N is the number of elements being removed. O(log(N)): zadd is a O(log(N)) command, where N is the number of elements already in the set. O(log(N)+M) zremrangebyscore : N is the number of total elements in the set and M is the number of elements to be removed. O(N+M*log(M)) sort 32

33 sort Sort the values within a list, set or sorted set. redis > rpush users:leto:guesses 5 9 10 2 4 10 19 2 (integer) 8 redis > sort users:leto:guesses 1) "2" 2) "2" 3) "4" 4) "5" 5) "9" 6) "10" 7) "10" 8) "19" redis 9> sadd friends:ghanima leto paul chani jessica alia duncan (integer) 6 redis > sort friends:ghanima limit 0 3 desc alpha 1) "paul" 2) "leto" 3) "jessica" Redis is single-thread, so sort on salve if large dataset Sort can store result to a key, one pattern for paginating through expensive sort results (millions of items, for example) is to save the result to a temporary key, set an expiry on it and use that for pagination via the LRANGE command. 33

34 Transaction Every Redis command is atomic, including the ones that do multiple things. – incr is essentially a get followed by a set – getset sets a new value and returns the original – setnx first checks if the key exists, and only sets the value if it does not – Msetnx fails if any key already exist (less important, now weve hash) MULTI command can run multiple commands as an atomic group. 1.Marks the start of a transaction block. Subsequent commands will be queued for atomic execution using EXEC.transactionEXEC DISCARD can be used in order to abort a transaction. In this case, no commands are executed and the state of the connection is restored to normal. DISCARD 2.The commands will be executed in order 3.The commands will be executed as a single atomic operation (without another client's command being executed halfway through) 4.That either all or none of the commands in the transaction will be executed redis > multi OK redis > hincrby groups:1percent balance -9000000000 QUEUED redis > hincrby groups:1percent balance -9000000000 QUEUED redis > exec 1) (integer) -9000000000 2) (integer) -18000000000 34

35 Optimistic locking using check-and- set WATCH / UNWATCH – keys are monitored in order to detect changes against them. If at least one watched key is modified before the EXEC command, the whole transaction aborts, and EXEC returns a Null multi-bulk reply to notify that the transaction failed.EXEC Null multi-bulk reply let's suppose Redis doesn't have INCRINCR WATCH mykey val = GET mykey val = val + 1 MULTI SET mykey $val EXEC Using WATCH to implement ZPOPWATCH WATCH zset element = ZRANGE zset 0 0 MULTI ZREM zset element EXEC 35

36 Expiration 1.Redis allows you to mark a key for expiration. 2.You can give it an absolute time in the form of a Unix timestamp (seconds since January 1, 1970) or a time to live in seconds. a.expire pages:about 30 -- delete key after 30 seconds b.expireat pages:about 1356933600 -- delete key at 12:00 a.m. December 31st, 2012. c.ttl pages:about -- check ttl d.persist pages:about -- remove expire limit e.setex pages:about 30 ' about us --set a string and specify a expire time 3. Lazy Expiration algorithm Keys are expired simple when some clients tries to access a key and the key is found to be time out 4. Once every second, I.Tests 100 random keys from expired keys set. II.Deletes all the keys found expired. III.If more than 25 keys were expired, it starts again from 36

37 What if no available memory Redis will return an error on write operations, but read-only query still works Can specify maxmemory to define a hard limit for memory usage maxmemory-policy: specify the algorithm to use when we need to reclaim memory a.volatile-lru (default) remove a key among the ones with an expire set, trying to remove keys not recently used. b.volatile-ttl remove a key among the ones with an expire set, trying to remove keys with short remaining time to live. c.volatile-random remove a random key among the ones with an expire set. d.allkeys-lru like volatile-lru, but will remove every kind of key, both normal keys or keys with an expire set. e.allkeys-random like volatile-random, but will remove every kind of keys, both normal keys and keys with an expire set. LRU and minimal TTL algorithms a) are not precise algorithms b) for default Redis will check three keys(maxmemory-samples) and pick the one that was used less recently 37 redis > set newkey "maxsize" (error) ERR command not allowed when used memory > 'maxmemory'

38 Redis Pipelining(How) Send multiple commands to the server without waiting for the replies at all, and finally read the replies in a single step. $ (echo -en "PING\r\nPING\r\nPING\r\n"; sleep 1) | nc localhost 6379 +PONG 38 Not paying the cost of RTT for every call; Client: INCR X Server: 1 Server: 2 Server: 3 Server: 4

39 Redis Pipelining(why we need) Redis is a TCP server using the client-server model and what is called a Request/Response protocol. – The client sends a query to the server, and reads from the socket, usually in a blocking way, for the server response. – The server processes the command and sends the response back to the client. 39 So for instance a four commands sequence is something like this: Client: INCR X Server: 1 Client: INCR X Server: 2 Client: INCR X Server: 3 Client: INCR X Server: 4 Network Round Trip: Latency?

40 Agenda Redis Manifesto Data structure : strings, hashes, lists, sets and sorted sets. Leveraging Redis Keys/O(n)/sort/expire/transaction/watch Redis Admin and maintenance Select database Monitor Redis Configure Persistence Starting a Redis Slave Handling a Dataset larger than memory Upgrade Redis BackUp Redis Sharding Redis Benchmarks The architecture of REDIS Cases 40

41 Databases in Redis 1.A database contains a set of data. 2.A database is to group all of an application's data together and to keep it separate from another application's. 3.databases are simply identified by a number with the default database being number 0. 4.Number of databases is set via databases param in config 5.change to a different database via select command – redis [1]> select 1 – OK – redis [1]> select 0 – OK – redis > 41

42 Monitor Redis 1.MONITOR, is actually part of the Redis replication system. If you telnet directly to Redis and type monitor, you'll see a live dump of all commands executing against the database. This is really useful for debugging. 2.config set slowlog-log-slower-than 0 3.Redis-stat : similar like prstat 4.Info command – redis > info – redis_version:2.4.8 – redis_git_sha1:00000000 – redis_git_dirty:0 – arch_bits:64 – multiplexing_api:epoll – gcc_version:4.1.2 – process_id:5898 – uptime_in_seconds:163519 – uptime_in_days:1 – lru_clock:1033766 – used_cpu_sys:1.19 – used_cpu_user:2.36 – used_cpu_sys_children:0.00 – used_cpu_user_children:0.00 – connected_clients:2 – connected_slaves:1 – client_longest_output_list:0 – client_biggest_input_buf:0 – blocked_clients:0 – used_memory:1384280 – used_memory_human:1.32M – used_memory_rss:3637248 – used_memory_peak:15032016 – used_memory_peak_human:14.34M – mem_fragmentation_ratio:2.63 – mem_allocator:jemalloc-2.2.5 – loading:0 – aof_enabled:0 – changes_since_last_save:0 – bgsave_in_progress:0 – last_save_time:1331542318 – bgrewriteaof_in_progress:0 – total_connections_received:757 – total_commands_processed:150064 – expired_keys:0 – evicted_keys:0 – keyspace_hits:50019 – keyspace_misses:3 – pubsub_channels:0 – pubsub_patterns:0 – latest_fork_usec:1875 – vm_enabled:0 – role:master – slave0:,31424,online – db0:keys=8,expires=0 – db1:keys=1,expires=0 42

43 Redis Admin and maintenance: Configure Persistence(1) Persistence Mode: snapshotting and AOF. It should be configured in a way that suits your dataset and usage patterns. 1.snapshotting, which consists of saving the entire database to disk in the RDB format (a compressed database dump). This can be done periodically at set times, or every time a configurable number of keys changes. # save save 900 1 --after 900 sec (15 min) if at least 1 key changed save 300 10 --after 300 sec (5 min) if at least 10 keys changed save 60 10000 --after 60 sec if at least 10000 keys changed 2.The alternative is using an Append Only File (AOF). This might be a better option if you have a large dataset or your data doesnt change very frequently../redis-server --appendonly yes 3.It is possible to combine both AOF and RDB in the same instance. 4.Master ->> Slave can be a option. 5.Both are sequential IO 6.When Redis starts, it will read RDB or AOF to load all data into memory. 43

44 Redis Admin and maintenance: Configure Persistence(2) 1.Snapshotting – performs point-in-time snapshots of dataset at specified intervals. ( – a full dump of your database to disk, overwriting the previous dump only if successful. – Can manually trigger snapshotting with the SAVE and BGSAVE commands. BGSAVE forks the main Redis process and saves the DB to disk in the background. SAVE performs the same operation as BGSAVE but does so in the foreground, thereby blocking your Redis server. – are also used when performing a master -> slave synchronization. 2.Append Only File(AOF) – keeps a log of the commands that change your dataset in a separate file. – an append only log. no seeks, nor corruption problems (redis-check-aof ) – Appendfsync: how often the AOF gets synched to disk (fsync syscall) : Always (be able to group commit), every sec, and no. – BGREWRITEAOF rewrites the AOF to match the current database; can reduce size of AOF greatly. (For example, if you are incrementing a counter 100 times, you'll end up with a single key in your dataset containing the final value, but 100 entries in your AOF. 99 of those entries are not needed to rebuild the current state.) 44

45 Redis Admin and maintenance: Master --Slave Use slave : Load balance read queires, standby, Backup,DW queries master-slave replication natively: A master can have multiple slaves. Slaves are able to accept other slaves connections. Redis replication is non-blocking on the master side, this means that the master will continue to serve queries when one or more slaves perform the first synchronization. configure replication on the configuration file before starting a server slaveof master-ip-or-hostname masterport masterauth master-password by connecting to a running server and using the SLAVEOF command. SLAVEOF master-ip-or-hostname [masterport] CONFIG SET masterauth password 45

46 Handling a Dataset Larger Than Memory Memory (VM) since version 2.0 (deprecated after Redis 2.4). vm-enabled yes vm-swap-file Allow a dataset bigger than your available RAM by swapping rarely used values to disk and keeping all the keys and the frequently used values in memory. 1.The keys are always kept in memory. Values can be swapped. 2.Redis server might end up blocking clients in order to fetch the values from disk. 3.Slow snapshot, Redis needs to read all the values swapped to disk in order to write them to the RDB file. AOF is better at this case. 4.VM also affects the speed of replication, because Redis masters need to perform a BGSAVE when a new slave connects. 5.SSDs such as Flash is encouraged 46

47 Upgrading Redis Redis cant do online binary upgrades solution 1.starting a new Redis server in slave mode, 2.switching over the clients to the slave 3.promoting the new server to the master role. make sure to test before doing it on your production servers. 47

48 Backing up Redis Depending on which Redis persistence model youre using. 1.With the default persistence model (snapshotting), youre best off using a snapshot as a backup. /* cold backup */ redis-cli BGSAVE Copy 2.If youre using only AOF, youll have to back up your log in order to be able to replay it on startup. BGREWRITEAOF regularly redis-check-aof --fix filename 3.backups on a slave Redis instance or Slave as a backup 48

49 Sharding Redis Where is Redis Cluster – Under development. Probably reasonable beta for summer 2012 and ship the first stable one before end of 2012. Have to implemented in the client library or application – you should probably use consistent hashing. – you will not be able to perform some operations that affect multiple keys, because those keys might be in different shards (servers). 49

50 Benchmarks--- How fast is Redis? redis-benchmark utility that simulates SETs/GETs done by N clients at the same time sending M total queries [hadoop@phxrueidb03 src]$./redis-benchmark -q -n 100000 PING (inline): 97370.98 requests per second PING: 101214.58 requests per second MSET (10 keys): 66357.00 requests per second SET: 105263.16 requests per second GET: 103199.18 requests per second INCR: 104493.20 requests per second LPUSH: 104931.80 requests per second LPOP: 104384.13 requests per second SADD: 104931.80 requests per second SPOP: 103950.10 requests per second LPUSH (again, in order to bench LRANGE): 104931.80 requests per second LRANGE (first 100 elements): 44964.03 requests per second LRANGE (first 300 elements): 22825.84 requests per second LRANGE (first 450 elements): 16564.52 requests per second LRANGE (first 600 elements): 12701.64 requests per second 50

51 Benchmarks- Redis VS memcached 51

52 Benchmarks--- How fast is Redis? Redis is a server: all commands involve network or IPC roundtrips. Cost of most operations is precisely dominated by network/protocol management. – low latency network Redis commands return an acknowledgment for all usual commands. Redis is an in-memory data store with some optional persistency options. Some persistency option would bring latency. – huge page & SSD Redis is a single-threaded server. It is not designed to benefit from multiple CPU cores. People are supposed to launch several Redis instances to scale out on several cores if needed. – Redis favors fast CPUs with large caches and not many cores. 52

53 Agenda Redis Manifesto Data structure : strings, hashes, lists, sets and sorted sets. Leveraging Redis Redis Admin and maintenance Configure Persistence/Redis Slave/Handling a Dataset larger than memory Upgrade Redis/BackUp Redis/Sharding Redis/ benchmarks The architecture of REDIS How Redis works Latency in Redis Memory efficiency in 2.2 Redis Security Cases 53

54 How Redis works How a command received by a client is processed internally by Redis: 1.Redis uses a single thread that manages synchronously all network connection. A thin event library has been implemented to abstract several unix system calls (epoll, select, kqueue). 2.Requests are managed with commands. Using a command table and according what event is read from sockets a command handler is invoked to perform desired action. 54

55 Latency in Redis Latency induced by network and communication – use aggregated commands (MSET/MGET) and Pipelining Single threaded nature of Redis – a mostly single threaded design ( I/O threads in background since 2.4) – all the requests are served sequentially Latency generated by slow commands – a request is slow to serve all the other clients will wait for this request to be served – commands operating on many elements, like SORT, LREM, SUNION and others. For instance taking the intersection of two big sets can take a considerable amount of time. – run all your slow queries on replciations Latency generated by fork – The fork operation (running in the main thread) can induce latency by itself. Latency induced by swapping (operating system paging) Latency due to AOF and disk I/O 55

56 Memory efficient for list adlist.h: A generic doubly linked list implementation typedef struct list { listNode *head; listNode *tail; void *(*dup)(void *ptr); void (*free)(void *ptr); int (*match)(void *ptr, void *key); unsigned int len; } list; typedef struct listNode { struct listNode *prev; struct listNode *next; void *value; } listNode; O(1) is cool but *prev/*next would take amounts of bytes if *value is few bytes. Ziplist (list-max-ziplist-entries 512 & list-max-ziplist-value 64) I.Save memory by using a little more CPU II.Pack list in a single block of memory III.Value header holds encoding / value length IV.O(memory size) LPUSH / LPOP V.Good fit for small payload, limited size 56

57 Memory efficient for hash Zmap (hash-max-zipmap-entries 512 & hash-max-zipmap-value 64) – keys and values are prefixed length "objects", the lookup will take O(N) where N is the number of elements in the zipmap and *not* the number of bytes needed to represent the zipmap. Other data structure also have similar improve ( sort sets, intset) 57

58 Skip-list for sort set 1.Consists of several levels, Each level is a sorted list 2.All keys appear in level 1 3.If key x appears in level n, then it also appears in all levels below n 4.An element in level n points (via down pointer) to the element with same key in the level below 5.Each level has int_min and int_max 6.Top points to the smallest element in the highest level 58

59 Memory structure lazy rehashing The more operation you run into an hash table that is rhashing, the more rehashing "steps" are performed, so if the server is idle the rehashing is never complete and some more memory is used by the hash table. active rehashing uses 1 millisecond every 100 milliseconds of CPU time in order to help rehashing the main Redis hash table (the one mapping top-level keys to values). 59

60 memory fragmentation Info used_memory:21279952 memory allocated to redis used_memory_human:20.29M used_memory_rss:23654400 memory from OS, result of ps or top used_memory_peak:21704152 used_memory_peak_human:20.70M mem_fragmentation_ratio:1.11 = used_memory_rss/used_memory mem_allocator:jemalloc-2.2.5 default in linux 2.4 and 2.6 String: dictEntry(12bytes)+sds(store key)+redisObject(12bytes)+sds(store value) 60 Set hello word = 16(dictEtnry) + 16 (redisObject) + 16(hello) + 16(world),

61 Redis Security 1.Redis is designed to be accessed by trusted clients inside trusted environments. Firewall on redis port 2.Redis is not optimized for maximum security but for maximum performance and simplicity. 3.Authentication feature The password is in clear text inside redis.conf file and client configuration AUTH command, like every other Redis command, is sent unencrypted 4.Data encryption support - None 5.Disabling of specific commands -- rename-command FLUSHALL "" 61

62 Agenda Redis Manifesto Data structure : strings, hashes, lists, sets and sorted sets. Leveraging Redis Redis Admin and maintenance The architecture of REDIS Event library /Memory efficient Latency /Security Cases a simple Twitter clone Sina Weibo 62

63 CASE 1: 1.List, lpush,ltrim show access history 2.Strings:incr show pageveiws 3.zset: show opt references, ref as a score 63

64 a simple Twitter clone Register: INCR global:nextUserId => 1000 SET uid:1000:username antirez SET uid:1000:password p1pp0 SET username:antirez:uid 1000 64 Circles and posts: uid:1000:followers => Set of uids of all the followers users uid:1000:following => Set of uids of all the following user uid:1000:posts => a List of post ids, every new post is LPUSHed here. Cookie: SET uid:1000:auth fea5e81ac8ca77622bed1c2132a021f9 SET auth:fea5e81ac8ca77622bed1c2132a021f9 1000 New post INCR global:nextPostId => 10343 SET post:10343 "$owner_id|$time|I'm having fun with Retwis" LPUSH to users followers foreach($followers as $fid) { $r->push("uid:$fid:posts",$postid,false); } Push to latest news: $r->push("global:timeline",$postid,false); $r->ltrim("global:timeline",0,1000); Pageing $posts = $r->lrange($key,$start,$start+$count); Making it horizontally scalable 1.Split by hash key

65 Sina weibo 65

66 Summary Redis Manifesto Memory #1;data structure server Data structure : strings, hashes, lists, sets and sorted sets, pub sub Leveraging Redis KEYS/Big O /Sort/EXPIRE/Transaction/Optimistic locking/Pipelining Redis Admin and maintenance Select /Redis/Persistence/Replication/ VM/Upgrade/BackUp/Sharding /Benchmarks The architecture of REDIS Event library /Memory efficient Latency /Security Cases / a simple Twitter clone /Sina Weibo 66

67 Redis Q&A 67

Download ppt "Open source, advanced key-value store, data structure server REmote DIctionary Server."

Similar presentations

Ads by Google