Home System Tutorial LINUX zetcd solves how to remove the dependence of applications on ZooKeeper

zetcd solves how to remove the dependence of applications on ZooKeeper

Mar 27, 2024 pm 10:50 PM
linux linux tutorial Red Hat linux system Secondary development linux command key value pair linux certification red hat linux linux video

Distributed systems usually rely on an arbitration system to work together. Generally, such systems use arbitration to ensure the accurate transmission of information to avoid split-brain. Such systems sacrifice versatility in exchange for ample design leeway, a practice that has clearly been exemplified by the proliferation of implementations. There are many such systems, such as: chubby, ZooKeeper, etcd and consul, etc. Although the concepts and protocols of these systems are different, they all provide similar key-value based distributed arbitration. As part of the plan to make etcd the most high-profile basic component of distributed systems, the etcd team has developed a new agent, zetcd, which allows the etcd cluster to consume ZooKeeper service requests without any changes.

ZooKeeper was the first open source implementation of arbitration software, which prompted it to become the preferred backend for many distributed systems. In theory, these systems should be compatible with etcd, but due to historical reasons, this is not the case; etcd clusters cannot replace ZooKeeper, and their data models and client protocols are incompatible with ZooKeeper applications; and vice versa. If the system is already in production, there is little incentive to plug it into a new backend and introduce new complexity.

Fortunately, the etcd v3 API is preparing to implement simulation support for the ZooKeeper data model through a standard proxy zetcd. zetcd is a new open source project developed by the etcd team. Today the first beta version of zetcd, v0, was released. 0.1, the goal is to manage and deploy the zetcd system in production systems.

zetcd The agent is deployed in front of the etcd cluster and serves a simulated ZooKeeper client port, allowing the ZooKeeper application to call etcd natively in the upper layer. Generally speaking, zetcd accepts ZooKeeper client requests, converts them into etcd's data model and API, forwards the requests to etcd, and then forwards the return information back in a way that the client can understand. The performance of zetcd is comparable to ZooKeeper, and it simplifies the management and operation complexity between ZooKeeper cluster and etcd. This article will reveal how to use zetcd, how zetcd works, and performance benchmarks.

Getting started with zetcd

What zetcd needs to run is a go compiler, source code obtained from the Internet, and a system that can run etcd. The following example shows how to obtain the zetcd source code and run the ZooKeeper command on zetcd. Since both etcd and zetcd are built based on the development branch, it is not recommended to do this in a production environment. This is just a simple example to explain how to use it.

First, obtain etcd and zetcd source code and compile it into binary code:

go get github.com/coreos/etcd/cmd/etcd 
go get github.com/coreos/zetcd/cmd/zetcd
Copy after login

Secondly, run etcd and connect zetcd to etcd client server:

#etcd uses localhost:2379 by default 
etcd & 
zetcd -zkaddr localhost:2181 -endpoints localhost:2379 &
Copy after login

Try zetd by adding a subscription and creating a key:

go install github.com/coreos/zetcd/cmd/zkctl 
zkctl watch / & 
zkctl create /abc "foo"
Copy after login

Conceptually, the above example completes adding a layer of zetcd to a single etcd instance.
zetcd solves how to remove the dependence of applications on ZooKeeper

Support for ZooKeeper in etcd3

In depth, zetcd will translate ZooKeeper's data model into an adapted etcd API. For key lookups, zetcd converts the ZooKeeper hierarchical directory into etcd's flat binary keyspace. To manage metadata, when writing to the etcd backend, zetcd utilizes memory-level transactions to securely and atomically update the information to ZooKeeper znode information.

ZooKeeper lists keys in directory mode (getChildren), while etcd uses interval (Range) mode. The following figure explains how zetcd encodes keys under etcd to effectively support listing in directory form. All zetcd keys in etcd have a prefix that includes the full directory name (for example: "/" and "/abc" represent depths of 0 and 1 respectively). To list a directory, zetcd issues a prefixed range request (for example, ["/zk/key/002/abc/", "/zk/key/002/abc0") to list the directories that satisfy the directory depth and path. All keys under /abc/. The depth limit only applies to the directory itself; if zetcd only uses the path but not the depth, etcd will return all keys in this directory, and zetcd will discard the result, otherwise only the keys in this directory will be returned.

zetcd solves how to remove the dependence of applications on ZooKeeper

Each ZooKeeper key has some metadata in the ZNode, that is, key adjustment, version and permissions, etc. Although etcd also has metadata for each key, it is much simpler than ZNode. For example, because there is no directory, there is no subversion, because etcd uses a role-based authentication mechanism, there is no ACL, and because the actual clock is out of scope, There is no timestamp. These additional metadata will be mapped to corresponding keys to describe a complete ZNode. When modifying metadata, zetcd uses memory-level soft transactions to automatically update a subset of keys, ensuring that ZNodes can remain consistent without expensive locking mechanisms.

In addition, zetcd can perform dynamic verification with an authorized ZooKeeper server. For comparison, zetcd can connect to both etcd and an external ZooKeeper server. When the client initiates a request to zetcd in this mode, the request will be forwarded to both zetcd and the ZooKeeper server. If the data responded by the two servers is inconsistent, zetcd will flag a warning for this response.

Performance Benchmark

由于数据的转换以及额外的网络开销,也许很容易觉得这样的模拟不切实际。尽管对于ZooKeeper或者etcd集群来说,zetcd有额外的花销,但是它仍然有一个优势,即某些etcd应用安装完毕后仍然需要ZooKeeper来协调的场景。例如,早期用户报告称在zetcd里通过etcd的TLS加密流量比一个类似的经典ZooKeeper配置更简单。在这些场景中,支持同一种ZooKeeper协议所带来的简单可靠性比性能更重要一些。 跟etcd性能工具的接口及报告形式类似,zetcd命令行工具zkboom可以用来判断一个zetcd的性能基准是否满足要求。其它ZooKeeper性能工具应该也可以用在zetcd;zkboom为用户提供了便利,我们不妨试试用它来做下创建key的测试:

go get github.com/coreos/zetcd/cmd/zkboom 
zkboom --conns=50 --total=10000 --endpoints=localhost:2181 create
Copy after login

zetcd应当可以为小负载提供充分的性能保障。一个简单两节点配置的延迟基准表明zetcd是可以承受大量请求的。具体配置为两台Linux服务器通过一个千兆交换机互联,其中一台设备在RAID磁盘配置上运行代理和和服务端,另外一台设备用于产生客户请求。zkbook通过创建一个空的key存储,然后从中读取128Kbytes的键值对来进行测量。用户请求被限制在每秒2500个请求,然后逐渐增加并发客户端数量。ZooKeeper 3.4.10和etcd结果对比见下图。

下图揭示了zetcd客户端并发数与创建key的平均延迟之间的关系。由于etcd在延迟上比ZooKeeper要有5-35ms的优势,zetcd有足够余地处理由于额外负载和网络跳转造成的延迟。比起ZooKeeper,zetcd代理始终还是存在20ms左右的差距,但是从处理2500请求的吞吐量数据来看,并没有出现队列堵塞。一种对zetcd写比较慢的解释是,与读不一样,由于数据模型上存在差异,所以在每个ZooKeeper key写时需要写多个key。

zetcd solves how to remove the dependence of applications on ZooKeeper

下图揭示了zetcd客户端并发数与key取值的平均延迟之间的关系。由于ZooKeeper的取值延迟比etcd要快那么2ms左右,想要zetcd提供数据的速度快过ZooKeeper的话恐怕还得依赖于etcd本身性能的进一步提升。然而,尽管zetcd需要从etcd请求额外的key来模拟Zookeeper znode的元数据,zetcd的命中延迟在等待etcd key提取数据方面只增加了大概1.5ms。zetcd在key的数据提取操作方面仅需一次往返,因为读请求会被打包到一个etcd事务中。

zetcd solves how to remove the dependence of applications on ZooKeeper
zetcd承诺上述性能基准测试的结果是合理的,在可接受的延迟情况下,可以轻松支撑每秒上千个操作。以上模拟对于Mesos,Kafka和Drill替代ZooKeeper提供了一个替代选择。但是对于zetcd本身来说性能方面仍有不少的提升空间。与此同时测试更多的ZooKeeper应用也会进一步推动zetcd成为ZooKeeper服务器的替代品。

zetcd从去年十月开始在开源社区发行,最近刚发布第一个版本:zetcd v0.0.1。尽管是第一个beta发行版,但是已经为未来生产系统提供稳定管理和部署。如果跟etcd配合起来使用,运行zetcd的系统将会一套自驱动的“ZooKeeper”集群,它可以自动后台升级,备份和TLS管理。

深入浅出学习etcd

etcd为分布式系统提供可靠、高效的配置管理服务,在Docker、Kubernetes、Mesos等平台中扮演了越来越重要的角色。作为2013年开始的项目,它还很年轻,官方文档中缺乏实现上全面、系统的介绍,本课程深入浅出地介绍了etcd的实现,并为运维和二次开发提供了系统的指导和建议。

The above is the detailed content of zetcd solves how to remove the dependence of applications on ZooKeeper. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

How to use docker desktop How to use docker desktop Apr 15, 2025 am 11:45 AM

How to use Docker Desktop? Docker Desktop is a tool for running Docker containers on local machines. The steps to use include: 1. Install Docker Desktop; 2. Start Docker Desktop; 3. Create Docker image (using Dockerfile); 4. Build Docker image (using docker build); 5. Run Docker container (using docker run).

How to view the docker process How to view the docker process Apr 15, 2025 am 11:48 AM

Docker process viewing method: 1. Docker CLI command: docker ps; 2. Systemd CLI command: systemctl status docker; 3. Docker Compose CLI command: docker-compose ps; 4. Process Explorer (Windows); 5. /proc directory (Linux).

Difference between centos and ubuntu Difference between centos and ubuntu Apr 14, 2025 pm 09:09 PM

The key differences between CentOS and Ubuntu are: origin (CentOS originates from Red Hat, for enterprises; Ubuntu originates from Debian, for individuals), package management (CentOS uses yum, focusing on stability; Ubuntu uses apt, for high update frequency), support cycle (CentOS provides 10 years of support, Ubuntu provides 5 years of LTS support), community support (CentOS focuses on stability, Ubuntu provides a wide range of tutorials and documents), uses (CentOS is biased towards servers, Ubuntu is suitable for servers and desktops), other differences include installation simplicity (CentOS is thin)

What to do if the docker image fails What to do if the docker image fails Apr 15, 2025 am 11:21 AM

Troubleshooting steps for failed Docker image build: Check Dockerfile syntax and dependency version. Check if the build context contains the required source code and dependencies. View the build log for error details. Use the --target option to build a hierarchical phase to identify failure points. Make sure to use the latest version of Docker engine. Build the image with --t [image-name]:debug mode to debug the problem. Check disk space and make sure it is sufficient. Disable SELinux to prevent interference with the build process. Ask community platforms for help, provide Dockerfiles and build log descriptions for more specific suggestions.

What computer configuration is required for vscode What computer configuration is required for vscode Apr 15, 2025 pm 09:48 PM

VS Code system requirements: Operating system: Windows 10 and above, macOS 10.12 and above, Linux distribution processor: minimum 1.6 GHz, recommended 2.0 GHz and above memory: minimum 512 MB, recommended 4 GB and above storage space: minimum 250 MB, recommended 1 GB and above other requirements: stable network connection, Xorg/Wayland (Linux)

vscode cannot install extension vscode cannot install extension Apr 15, 2025 pm 07:18 PM

The reasons for the installation of VS Code extensions may be: network instability, insufficient permissions, system compatibility issues, VS Code version is too old, antivirus software or firewall interference. By checking network connections, permissions, log files, updating VS Code, disabling security software, and restarting VS Code or computers, you can gradually troubleshoot and resolve issues.

Detailed explanation of docker principle Detailed explanation of docker principle Apr 14, 2025 pm 11:57 PM

Docker uses Linux kernel features to provide an efficient and isolated application running environment. Its working principle is as follows: 1. The mirror is used as a read-only template, which contains everything you need to run the application; 2. The Union File System (UnionFS) stacks multiple file systems, only storing the differences, saving space and speeding up; 3. The daemon manages the mirrors and containers, and the client uses them for interaction; 4. Namespaces and cgroups implement container isolation and resource limitations; 5. Multiple network modes support container interconnection. Only by understanding these core concepts can you better utilize Docker.

What is vscode What is vscode for? What is vscode What is vscode for? Apr 15, 2025 pm 06:45 PM

VS Code is the full name Visual Studio Code, which is a free and open source cross-platform code editor and development environment developed by Microsoft. It supports a wide range of programming languages ​​and provides syntax highlighting, code automatic completion, code snippets and smart prompts to improve development efficiency. Through a rich extension ecosystem, users can add extensions to specific needs and languages, such as debuggers, code formatting tools, and Git integrations. VS Code also includes an intuitive debugger that helps quickly find and resolve bugs in your code.

See all articles