• Redis Sentinel Error Connection Reset By Peer
• Redis Sentinel Reset Tool
• Redis Sentinel Connection Reset By Peer

VM Instance Name Apps / Services IP Address; Instance 01: haproxy-01: HAProxy: 192.168.1.10: Instance 02: redis-01: Redis, Sentinel (Master) 192.168.1.11: Instance 03. # Host and port we will listen for requests on bind 127.0.0.1 port 16380 # # 'redis-cluster' is the name of our cluster # # each sentinel process is paired with a redis-server process # sentinel monitor redis-cluster 127.0.0.1 6380 2 sentinel down-after-milliseconds redis-cluster 5000 sentinel parallel-syncs redis-cluster 1 sentinel failover.

An Introduction To Redis

Redis is an open-source in-memory data structure store, which is a very popular selection among developers due to its variety of data structures that satisfy all application needs. The Remote Dictionary Server or Redis, is mainly implemented for caching purposes, as a message broker and also utilized as a NoSQL Key-Value database for different use cases. It is also a BSD licensed open-source, in-memory data structure store – where data is not written to disk but kept in volatile RAM. Product engineering and software development companies have found great potential in Redis as its advanced capabilities in modern applications enables search, messaging, streaming, graph, and other innovative capacities. It is also used as a database, cache, and message broker.

Therefore, whether you are a developer interested in learning Redis or a professional thinking about implementing Redis in your enterprise, our article can provide exactly what you’re looking for! So, let’s begin!

What Are The Features of Redis?

Often referred to as the “Leatherman of Databases”, the simple yet flexible design philosophy of Redis makes it an effective choice in solving a multitude of demanding data processing tasks for software development companies. Redis data structures resolve very complex programming problems with simple commands executed within the data store, reducing coding effort, increasing throughput, and reducing latency. Here are a few features of Redis!

Fast: Redis is extremely fast as it stores the whole dataset in primary memory.

Persistence: Based on the number of updates or elapsed time, data can be asynchronously saved to disk.

Data replication: Master-slave cache nodes can be set up. Slave node always listens to master i.e. when the master node is updated, the slave is updated automatically.

Clients available in all popular programming languages: Redis client API is developed in almost all popular languages like C, C++, Java, JavaScript, Python, etc.

Lightweight: Redis is written in ANSI C, with no external dependencies.

Snapshotting: Redis by default stores snapshots of your data in dumb.rdb file. In the case of server restart, Redis will load the data from a backup file and put them into memory.

High Availability: To ensure high availability of data, there is a built-in support for asynchronous master/slave replication, for which Redis Sentinel can be used.

What Is Redis Sentinel?

Redis Sentinel is the high availability solution offered by Redis, . In case of a failure in your Redis cluster, Sentinel will automatically detect the point of failure and bring the cluster back to stable mode without any human intervention. In other words, Redis Sentinel is a system that can resist Redis deployment without human intervention.

Capabilities of Redis Sentinel

Caching is a technique used to accelerate application response times and help applications scale by placing frequently needed data very close to the application. Redis, an open-source, in-memory, data structure server is frequently used as a distributed shared cache (in addition
to being used as a message broker or database) because it enables true statelessness for an applications’ processes while reducing duplication of data or requests to external data sources. Other than caching, here are a few other capabilities of Redis Sentinel.

Monitoring: Check all instances i.e. whether master and slave are working properly.

Notification: If one of the Redis instances are not working as expected, Sentinel can notify system administrators or another program through an API.

Automatic failover: If the master is not working as expected, Sentinel promotes one of the slaves as master and then make the additional slaves use the new master.

Sentinel is the source of authority for clients. The client connects to Sentinel for asking the address of the current Redis master.

How Does A Sentinel Work

The idea of Sentinel is that it’s an independent node that keeps track of the master node and other slave nodes. In a master-slave scenario, the master is meant for writing and slaves are for reading. When the master goes down, a slave node will become a master and it will have the capability to do both reading and writing. All other slaves will become slaves to the new master.

A Simple Example On Redis Sentinel In Windows

The Redis project does not officially support windows. However, Redis has been ported to Windows (64 bit) by Microsoft Open Technology Project. The default installation folder is ‘C:Program FilesRedis’.

This example consists of 3 Redis instances and one Sentinel instance. For the sake of easiness, all these instances are hosted in one machine itself.

Redis_Master = Master instance

Redis_Slave1 = Slave 1 instance i.e. slave of Redis_Master

Redis_Slave2 = Slave 2 instance i.e. slave of Redis_Master

Sentinel = Sentinel instance

To deploy 3 Redis instances, create three copies of ‘redis.windows.conf’ file in ‘C:Program FilesRedis’ as below

redis.windows-service-redis1.conf

• Give port as 6379 (default port of Redis)

redis.windows-service-redis2.conf

• Given port as 6380
• Replace slaveof <masterip> <masterport> with slaveof 127.0.0.1 6379

redis.windows-service-redis3.conf

• Given port as 6381
• Replace slaveof <masterip> <masterport> with slaveof 127.0.0.1 6379

To deploy Sentinel instance, create sentinel.conf file in ‘C:Program FilesRedis’ as below

sentinel.conf

port 26379

sentinel monitor Redis_Master 127.0.0.1 6379 1

sentinel down-after-milliseconds Redis_Master 30000

sentinel parallel-syncs Redis_Master 1

sentinel failover-timeout Redis_Master 180000

The syntax for above configuration is :

sentinel monitor MasterName RedisMasterServerIP RedisMasterPort Quorum

sentinel down-after-milliseconds MasterName DownTimeout

sentinel failover-timeout MasterName FailoverTimeout

sentinel parallel-syncs MasterName ConcurrentServerSync

where

Line 1 – tells the master to be monitored. The quorum is used to detect failures. When the master is not reachable, the number of sentinels in the quorum should agree about it.

Line 2 – master will be considered as down when it’s not reachable for the Sentinel for the time specified in milliseconds.

Line 3 – The timeout in milliseconds indicates the time that Sentinel will wait after a failover before starting a new failover.

Line 4 – The number of slaves that can synch with the new master at the same time after failover.

Let’s create 3 Redis instances and one Sentinel instance as windows service. The below commands needs to be run in command prompt (Run as administrator) to do the same:

Redis Master and Slaves

• sc create Redis_Master DisplayName= “Redis_Master” binPath= “”C:Program FilesRedisredis-server.exe” –service-run ”C:Program FilesRedisredis.windows-service-redis1.conf”

• sc create Redis_Slave1 DisplayName= “Redis_Slave1” binPath= “”C:Program FilesRedisredis-server.exe” –service-run ”C:Program FilesRedisredis.windows-service-redis2.conf”

• sc create Redis_Slave2 DisplayName= “Redis_Slave2” binPath= “”C:Program FilesRedisredis-server.exe” –service-run ”C:Program FilesRedisredis.windows-service-redis3.conf”

Redis Sentinel : sc create Redis_Sentinel DisplayName= “Redis_Sentinel” binPath= “”C:Program FilesRedisredis-server.exe” –service-run ”C:Program FilesRedissentinel.conf” –sentinel

Start Master, Slaves and Sentinel services respectively from Windows Services (services can be opened by through run [windows key + R] by giving services.msc)

The status of Sentinel can be checked through redis-cli using given below command:

redis-cli -p 26379 info sentinel

As you see, Sentinel is monitoring ‘Redis_Master’ and its status is Ok.

To know the current master, the following command can be used:

redis-cli –p 26379 sentinel get-master-addr-by-name Redis_Master

Now stop Redis_Master service from Windows Services. Execute the above command again after 30 seconds. You can see that one of the slaves is re-configured as Master.

Points To Be Considered Before Deploying Redis Sentinel

1. For a robust deployment, at least three sentinel instances are needed.
2. The three instances should be in different computers or virtual machines that are believed to be failed in independently.
3. Sentinel should be supported in the client library.
4. Docker or other forms of port mapping should be mixed with care as it will break the auto-discovery of other sentinels and a list of slaves.

Redis Sentinel Error Connection Reset By Peer

Conclusion

As we all know, software processes are made of data; every process’ instructions, inputs, runtime state and outputs are data stored somewhere. Whether high-level smart IoT services, web development services or product engineering services, enterprises require a better database, cache, and message broker for processing. Redis is designed to be accessed by trusted clients inside trusted environments and provides quick processing as well as help speed up response times through caching data. Contact us today, if you are willing to scale up your business; we offer Redis hosting and fully managed solutions on a cloud of your choice.

Sentinel role

Sentinel is a distributed system, which guarantees the high availability of programs. It is used to monitor any number of master servers and all slave servers under these master servers. In case of failure, it selects a new master through voting mechanism and connects all slaves to the new master.

monitor

Continuously check whether the master and slave are running normally, master survivability detection and master and slave operation detection.

notice

When there is a problem with the monitored server, send a notification to other (sentinel, client).

Automatic fail over

Disconnect the master from slave, select a slave as the master, connect other slave to the new master, and inform the client of the new server address.

be careful

Sentry is also a redis server, but it does not provide data service. Usually, the number of sentries is odd

Activate the sentry

configuration file

Sentinel default profilesentinel.conf

General withsentinel_port.confName the Sentinel’s profile

configuration information

start-up

Configure master-slave structure, taking 1master 2 slave as an example.

1 start master and slave first

Please refer to the master-slave blog for master-slave configurationMaster and slave

2. Activate the sentry

Sentinel command

PING：PONG
Sentinel masters: lists all monitored master servers and their current status.
Sentinel Slaves: lists all slave servers of a given master server and their current status.
Sentinel get master addr by name: returns the IP address and port number of the master server with the given name. If the primary server is performing a fail over operation, or the fail over operation for the primary server has been completed, this command returns the IP address and port number of the new primary server.
Sentinel reset: resets all primary servers whose names match the given pattern. The pattern parameter is a global style pattern. The reset operation makes clear all the current states of the master server, including the failure transition in progress, and removes all the discovered and associated slave servers and sentinel of the master server.
Sentinel fail over: when the primary server fails, it is forced to start an automatic fail over without asking other sentinels for advice (however, the sentinel that initiated the fail over will send a new configuration to other sentinels, and other sentinels will update it accordingly).

Initialize Sentinel

initialize server

As can be seen from the following startup code, the startup mode is determined by the functioncheckForSentinelModeTo decide whether to use sentinelMode of a start, add the command is also usedsentinelcmdsCommand list for

From the source code, we can see that there are two ways to start sentry

Either way, it startsredisIt will be implementedinitServerConfigThe difference isSentinelIt will also be implementedinitSentinelConfig、initSentinelTwo initialization functions. Let’s see what these two functions do.

Replace Sentinel’s private code

initSentinelConfig()This function will useSentinelThe configured properties override the server’s default properties.

initSentinel()A command table will be loaded. A main query commandINFOIt is also different from ordinary server, but uses a special version.

Initialize sentinel state

After the command table is loaded, it is immediately followed bysentinelStateandsentinelRedisInstance An initialization of the structure.

SentinelIn statemastersThe dictionary records all the monitored master server information. The key is the name of the server and the value is the corresponding value of the monitored master serversentinel.c/sentinelRedisInstanceStructure. eachsentinelRedisInstanceThe instance structure represents a monitorRedisServer instance, which can be a master server, a slave server, or another sentinel server.

aboutsentinelStateThe initialization of themastersThe initialization of the master dictionary is based on the inputsentinelConfiguration file（sentinel_26379.conf）To do it. It is mainly monitoredmasterOfipand port。

be carefulThese are maintained and used by sentinel.

sentinelState

sentinelRedisInstance

name
The name of the instance
The name of the master server is set by the user in the configuration file
The name of the slave server and sentinel is automatically set by sentinel
The format is IP: port, for example, “127.0.0.1:26379”

runid
Run ID of the instance

sentinelAddr
The address of the instance

Properties specific to the primary server instance

sentinels
Others also monitor all sentinels of this master server

slaves
If this instance represents a master server
Then this dictionary holds the slave server under the master server
The key of the dictionary is the name of the slave server, and the value of the dictionary is the sentinelredisinstance structure corresponding to the slave server

quorum
The number of support votes needed to judge this instance as objective down

parallel_syncs
SENTINEL parallel-syncs The value of the option
The number of slave servers that can synchronize the new master server at the same time when performing a fail over operation

auth_pass
The password required to connect the master and slave servers

From server instance specific properties

master_link_down_time
The time when the master-slave server was disconnected

slave_priority
Slave priority

slave_reconf_sent_time
When performing a fail over operation, slaveof is sent from the serverTime of command

master
Instance of master server (used when this instance is a slave server)

slave_master_host
The master server IP recorded in the reply to the info command

slave_master_port
The main server port number recorded in the reply to the info command

slave_master_link_status
The master-slave server connection status recorded in the reply to the info command

slave_repl_offset
Copy offset from server

In structuresentinelAddrIt holds the address and port of the object.

Make a connection

Sentinel will connect firstsentinelin mastersEvery one of themmaster, and in everymasterandsentinelCreate two betweenAsynchronous connectionOneCommand connectionOneSubscribed Links . At this time, sentinel will become the client of the master. It can send commands to the master server and get relevant information from the command reply.

Command connection

It is specially used to send commands to the master server and receive command replies. For example, sentinel sendsINFOOrders.

Subscription connection

Dedicated to subscriber master_sentinel_:helloChannel. such assentinelTo the Lord, to the slave, to the othersentinelsend outsentinelAnd the main database information.

In the publish and subscribe function of redis, the information sent will not be saved in the redis server. If the client that needs to receive the message is not online or disconnected when it arrives, the client will lose the information. In order not to lose_sentinel_:helloAny information about the channel,sentinelYou must use a dedicated subscription connection to receive information from that channel.

Get master server information

SentinelBy default, every10Second frequency to the main serverINFOCommand to get the current information of the master server by analyzing the command reply. Sentinel can obtain the following two aspects of information:

1 information about the master server itself, including the server run_ ID, the server role of the role.

2 corresponding to the main serverAll slave serversFrom the server IP and port.

Get information from server

WhenSentinelWhen a new slave server appears,SentinelIn addition, the corresponding instance structure will be created for the new slave server（sentinelRedisInstance）In addition, aCommand connectionandSubscription connection。

SentinelIt will still send one every 10 seconds, just like the main serverINFOCommand to get the current information from the server.

run_ id、role、ip、port 、master_ link_ Status, slave_ Priority (priority of the slave server).

Send information to master and slave servers

By default,SentinelIt will send commands in the following format to all monitored master servers and slave servers through command connection once every 2 seconds:

This command is sent to the_sentinel_:helloThe channel sends a message. The content of the message consists of several parameters

(1) s_The first parameter recordssentinelIt’s a message of its own.

(2) m_The first parameter records the information of the master server. If sentinel is monitoring the master server, then these parameters are the information of the master server. If sentinel is monitoring the slave server, then these parameter records are the information of the master server being copied by the slave server.

for example

Receive channel information from master and slave servers

WhenSentinelAfter establishing a subscription connection with a master or slave server,SentinelIt will passSubscription connectionSend to serversubscribe_sentinel_:hello 。

For each and everySentinelConnected server,SentinelConnect to the server by command_sentinel_:helloChannel sends information, and then through the subscription connection from the server’s_sentinel_:helloThe channel receives information.

So when there’s a new oneSentinelWhen you connect in, send thesubscribe_sentinel_:helloBy existingSentinelReceive (I will also receive this message from myself).

Be aSentinelfrom_sentinel_:helloWhen the channel receives a message,SentinelI will analyze this information and extract IP, port and run in the information_ ID and other 8 parameters, and check the following: if this message is sent by itself, it will be ignored directly. If it’s newSentinelIn this case, sentinel will update the structure of the corresponding primary server instance, that is, the newly addedSentinelAdd tosentinelsIn the dictionary.

eachSentinelEach has its own onesentinelsDictionaries,SentinelsDictionary information holds all information except itselfSentinelInformation.

Offline status

There are two different concepts about offline in sentinel of redis: (1) subjectively down (sdown for short) refers to the offline judgment made by a single sentinel instance to the server, and there will be no failover at this time. (2) Objective down (owdown) means that when multiple sentinel instances make sdown judgment on the same server, the target sentinel will fail over the primary server. This article introduces in detail.

Subjective offline state

defaultSentinelWill be sent once per second to all instances (including master, slave, other sentinel) with which it creates a command connectionPINGCommand, and judge whether the instance is online by instance reply.

Legal response

+pong 、 -loading、 -masterdown。

Redis Sentinel Reset Tool

Invalid reply

In addition, all replies or no replies are considered invalid. No reply means no reply within a specified time.

The value of the down after milliseconds option set by the user is not only used by sentinel to determine the subjective offline status of the master server, but also used to determine the subjective offline status of all slave servers under the master server and other sentinel monitoring the master server.

theremasterIs the name of the primary server, and the port is the default6379 ，2representativesentinelThere are many problems in the cluster2individualsentinelthinkmaster Only when the status is offline, can we really think that the master is not available (that isObjective offline)。

It’s not just going to besentineljudgemasterEnter the subjective offline standard, but also to judge allSlave Library, otherssentinelThe standard of entering subjective offline.

Redis Sentinel Connection Reset By Peer

When multiple sentinels are set, the subjective offline duration may be different

For multiplesentinelWhen monitoring the same primary server together, thesesentinelIn the configuration filesentinle.confSet indown-after-millisecondsValues can also be different, so when asentinelWhen the main server is judged as subjective offline, the othersentinelYou may still think that the primary server is online. Only all of themsentineOnly when they have entered the subjective offline state, can they be regarded as the main channelmasterIn the subjective offline state.

Objective offline status

WhenSentinelAfter a master server is judged to be offline subjectively, in order to confirm whether the master server is offline or not, it will monitor other servers of the master serverSentinelWhen there are more than half (depending on the specific configuration, generally more than half), for examplesentinel monitor mymaster 127.0.0.1 6379 2In the middle2When a judge is offline, it is considered that he is offline objectively.)

WhenmasterAfter being determined to be offlinesentinelWe’ll pick onedecision makerTo perform a fail over operation. Objective offline conditionsPrimary server only。

is-master-down-by-addrThe command is used to determine whether to go offline objectively

sentinelThe current era of configurationcurrent_epochFor electiondecision maker sentinel, run_idIt can be * or the running ID of sentinel.

Decision maker selection

Suppose there are foursentinelThese foursentinelBoth voters and candidates (these four must be healthy).

1 cannot have one of the following three Tags: Sri_ S_ DOWN SRI_ O_ DOWN SRI_ DISCONNECTED

The heart rate is normal

3 priority cannot be 0 (slave > slave)_ priority）

4. Info data cannot time out

5. The disconnection time of master-slave connection cannot be timeout

The process of voting is very simple, every one of themsentinelAll will be their ownip、 port、current_epoch、run_idfromis-master-downSend tohelloChannel.

sentinelWho was the first to get itis-master-downInformation, vote for the correspondingsentinel。

After onecurrent_epochThe largest, and more than half. Is selected asdecision makerOtherwise, another round, every additional roundcurrent_epoch + 1Until the election.

Fail over

Select candidate slave

1 Online

2 fast response

3 the shortest time to disconnect from the original master

4. Priority principle

Priority > offset > runid

Finally, a new one is selectedmasterAnd then to the newmaster send out

Then declare the newmaster

Finally, the original master is used as the slave. When you go back online,sentinelWill sendsalveofThe command makes it a slave.

summary

• sentinelIt’s just a computer running in a special moderedisServer, which uses a different command table from the normal mode, as well as different commands from the normal mode.

• sentinelSend to primary serverINFOCommand to obtain the address information of all the slave servers under the master server, and create the corresponding instance structure for these slave servers, as well as the command connection and subscription connection to these slave servers.

• Normally,sentinelTo the monitored master and slave servers every 10 secondsINFOCommand, when the primary server is offline, orsentinelWhile the primary server is being failed over,sentinelSend to slave serverINFOThe frequency of the command will be changed to once a second.

• For multiple servers that monitor the same master and slave serversentinelIn other words, they will send messages to the monitored_sentinel_:helloChannel sends messages to other channelssentinelDeclare your own existence.

• eachsentinelI’ll get it from you, too_sentinel_:helloReceive others in medium channelsentinelSend information, and based on this information for other userssentinelCreate the corresponding instance structure and command connection.

• sentinelOnly command and subscription connections are created with master and slave servers,sentinelAndsentinelOnly command connections are created between.

• sentinelSend data to the instance (including master, slave and others) once per secondsentinel）SendPINGCommand, and judge whether the instance is online according to the reply of the instancesentinelWhen an invalid reply is sent,sentinelThis instance will be judged as subjective offline.

• WhensentinelWhen a master server is judged to be subjectively offline, it will monitor other users of the master server to the same serversentinelAsk them if they agree that the master server has entered the subjective offline state.

• WhensentinelAfter collecting enough subjective offline votes, it will judge the primary server as objective offline and initiate a failover operation against the primary server.