Distributed Cache Redis HyperLogLog Learning Tutorial Explanation and Example Analysis

January 15, 2023

The cardinality estimation algorithm is to use the accuracy to exchange space. To illustrate this, we use three different calculation methods to count the number of different words in all Shakespeare's works. Note that our input dataset adds extra data to be higher than the reference base of the problem. The three technologies are: Java HashSet, Linear ProbabilisTIc Counter, and a Hyper LogLog Counter. The results are as follows:

分布式缓存Redis之HyperLogLog学习教程讲解

The table shows that we only use 512 bytes for these words, and the error is within 3%. In contrast, HashMap has the highest count accuracy, but requires nearly 10MB of space, so you can easily see why cardinality estimates are useful. Accuracy is not very important in practical applications. It is a fact that using probability counters can save a lot of space in most network sizes and network calculations.

Furthermore, if we want to implement a feature that records the number of independent IPs that the website visits each day:

Collection implementation:

Use the collection to store the IP of each visitor, get the multiple independent IPs by the nature of the collection (each element in the collection is different), and then derive the number of independent IPs by calling the SCARD command.

For example, the program can use the following code to record the IP of each website visitor on December 5, 2017:

Ip = get_vistor_ip() SADD'2017.12.5::unique::ip'ip

Then use the following code to get the unique IP number for the day:

SCARD'2017.12.5::unique::ip'

Collection implementation problem

Using a string to store each IPv4 address can take up to 15 bytes (in the format 'XXX.XXX.XXX.XXX', such as '202.189.128.186').

The following table shows the amount of memory that is required to use a collection to record a different number of independent IPs:

The number of independent IPs a day, one month a year, one million 15 MB 450 MB 5.4 GB
10 million 150 MB 4.5 GB 54 GB
100 million 1.5 GB 45 GB 540 GB

As the IP of the collection records increases, more and more memory is consumed. Also, if you want to store an IPv6 address, you will need more memory. To better address issues like independent IP address calculations, Redis added the HyperLogLog structure in version 2.8.9.

Redis data structure HyperLogLog

Redis HyperLogLog is an algorithm used to perform cardinal statistics. The advantage of HyperLogLog is that when the number or size of input elements is very large, the space required to calculate the cardinality is always fixed and small. In Redis, each HyperLogLog key takes only 12 KB of memory to calculate the cardinality of nearly 2^64 different elements. This is in stark contrast to the more elements that consume more memory when using the set to calculate the cardinality. However, because HyperLogLog only calculates the cardinality based on the input elements, not the input elements themselves, HyperLogLog cannot return the input elements as a collection.

What is the cardinality?

For example, the data set {1, 3, 5, 7, 5, 7, 8}, then the base set of this data set is {1, 3, 5, 7, 8}, and the cardinality (non-repeating element) is 5. The cardinality estimate is to quickly calculate the cardinality within the acceptable range of the error.

Estimate: The cardinality given by the algorithm is not exact and may be slightly more or slightly less than actual, but it will be controlled within a reasonable range.

Several orders

Add elements to HyperLogLog

1, PFADD key element [element ...]

Add any number of elements to the specified HyperLogLog.

This command may modify the HyperLogLog to reflect the new cardinality estimate. If the base estimate of HyperLogLog changes after the command is executed, the command returns 1 otherwise returns 0.

The complexity of the command is O(N) and N is the number of elements added.

2, PFCOUNT key [key ...]

Returns the cardinality estimate for a given HyperLogLog.

When only one HyperLogLog is given, the command returns the cardinality estimate for the given HyperLogLog.

When multiple HyperLogLogs are given, the command will first calculate the union of the given HyperLogLog, and then get a merged HyperLogLog, and then return the cardinality estimate of the merged HyperLogLog as the result of the command (the merged HyperLogLog will not It is stored and will be deleted after use).

When the command acts on a single HyperLogLog, the complexity is O(1) and has a very low average constant time.

When the command is applied to multiple HyperLogLogs, the complexity is O(N) and the constant time is much larger than when processing a single HyperLogLog.

Custom Slip Ring

What is a slip ring?

A slip ring is an electromechanical device that allows electrical current to flow through a rotating assembly. It is often used in conjunction with a commutator to create an electrical connection in a rotating system. The device consists of a number of contact rings mounted on the rotating assembly and two or more stationary contacts. When the rotating assembly is rotated, the contact rings pass by the stationary contacts, creating an electrical connection.


Types of slip rings

A slip ring is an electromechanical device that allows the transmission of power and electrical signals through a rotating assembly. Slip rings are commonly used in applications where it is not possible or practical to make a permanent electrical connection, such as on a rotating antenna, wind turbine, or radar dish. There are many different types of slip rings available, each with its own set of benefits and drawbacks. Some of the more common types of slip rings include fiber optic slip rings, rotary unions, and contactless slip rings.


Advantages of custom slip rings

When it comes to slip rings, custom-made ones have a lot of advantages over the standard, off-the-shelf variety. Custom slip rings are designed specifically for your application, so you can be sure they`ll meet all your needs. They`re also made to order, so you can get exactly the features you want. And because they`re made specifically for you, they`re often less expensive than generic slip rings.


Disadvantages of custom slip rings

In many industrial and commercial applications, a custom slip ring is the best way to transfer power and/or data between two systems. While custom slip rings offer many advantages, they also have some disadvantages. In this article, we will discuss the disadvantages of custom slip rings.


One disadvantage of custom slip rings is that they are often more expensive than other types of electrical connectors, such as wire harnesses or connectors. Additionally, custom slip rings can be difficult to install and may require specialized tools. Another disadvantage is that custom slip rings are not always as reliable as other types of electrical connectors. They can fail prematurely if not properly maintained or used in the correct application.


Conclusion: When to use a custom slip ring


A custom slip ring is an electromechanical device that allows for the transmission of electrical signals and power between rotating and stationary components. The use of a custom slip ring can be beneficial in a number of ways, including improved performance, increased efficiency, and longer life. In some cases, a custom slip ring may be the only option available for achieving the desired results. When considering a custom slip ring as an option for your application, it is important to understand the different factors that will impact its performance.


Custom Slip Ring,Rotary Joint Price,High Pressure Rotary Joint,Moflon Rotary Union

Dongguan Oubaibo Technology Co., Ltd. , https://www.sliprob.com