Less is more's Blog - 设计模式、算法与分布式系统专业技术博客

Netty 完全指南：构建高性能网络应用的终极秘籍

lesshash@blog.lesshash.com/ (lesshash) — Thu, 18 Sep 2025 15:55:00 +0800

一、为什么选择 Netty？

1.1 网络编程的挑战

在现代应用开发中，网络编程面临诸多挑战：

高并发连接：需要同时处理成千上万的连接
低延迟要求：毫秒级的响应时间
复杂协议处理：HTTP、WebSocket、自定义协议
资源管理：内存泄漏、线程池管理
跨平台兼容性：Windows、Linux、macOS

1.2 Netty 的优势

Netty 是一个高性能、异步的网络应用框架，为快速开发可维护的高性能网络服务器和客户端提供了强大支持。

核心优势

🚀 高性能：基于 Java NIO，零拷贝、内存池等优化
🎯 低延迟：事件驱动的异步架构
💪 高并发：单机支持数万甚至数十万连接
🔧 易用性：简洁的 API 设计，丰富的编解码器
🛡️ 稳定性：久经考验，大量知名项目使用

性能表现

让我们看看 Netty 与其他框架的性能对比：

从上图可以看出，Netty 在各种响应大小下都表现出色，特别是在小数据包场景下性能优势明显。

在延迟方面，Netty 同样表现优异，P99 延迟控制在 3ms 以内。

二、核心概念详解

2.1 EventLoop 和 EventLoopGroup

概念理解

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


// EventLoop 是 Netty 的核心抽象
public class EventLoopExample {
 public static void main(String[] args) {
 // Boss EventLoopGroup：负责接受连接
 EventLoopGroup bossGroup = new NioEventLoopGroup(1);

 // Worker EventLoopGroup：负责处理 I/O 事件
 EventLoopGroup workerGroup = new NioEventLoopGroup();

 try {
 ServerBootstrap b = new ServerBootstrap();
 b.group(bossGroup, workerGroup)
 .channel(NioServerSocketChannel.class)
 .childHandler(new ChannelInitializer<SocketChannel>() {
 @Override
 public void initChannel(SocketChannel ch) {
 // 配置 pipeline
 }
 });

 System.out.println("Boss EventLoopGroup 线程数: " +
 ((NioEventLoopGroup) bossGroup).executorCount());
 System.out.println("Worker EventLoopGroup 线程数: " +
 ((NioEventLoopGroup) workerGroup).executorCount());

 } finally {
 workerGroup.shutdownGracefully();
 bossGroup.shutdownGracefully();
 }
 }
}

EventLoop 线程模型

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


// 自定义 EventLoop 配置
public class EventLoopConfiguration {

 public static EventLoopGroup createOptimizedEventLoopGroup() {
 // 根据 CPU 核数确定线程数
 int threads = Runtime.getRuntime().availableProcessors() * 2;

 return new NioEventLoopGroup(threads, new ThreadFactory() {
 private final AtomicInteger counter = new AtomicInteger(0);

 @Override
 public Thread newThread(Runnable r) {
 Thread thread = new Thread(r);
 thread.setName("Netty-Worker-" + counter.incrementAndGet());
 thread.setDaemon(true);
 // 设置线程优先级
 thread.setPriority(Thread.MAX_PRIORITY);
 return thread;
 }
 });
 }

 // EventLoop 任务调度示例
 public static void scheduleTask(EventLoop eventLoop) {
 // 延迟执行任务
 eventLoop.schedule(() -> {
 System.out.println("延迟任务执行: " + Thread.currentThread().getName());
 }, 5, TimeUnit.SECONDS);

 // 周期性任务
 eventLoop.scheduleAtFixedRate(() -> {
 System.out.println("周期性任务: " + new Date());
 }, 0, 10, TimeUnit.SECONDS);
 }
}

2.2 Channel 和 ChannelHandler

Channel 生命周期

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66


// 自定义 ChannelHandler 演示完整生命周期
public class LifecycleChannelHandler extends ChannelInboundHandlerAdapter {

 @Override
 public void handlerAdded(ChannelHandlerContext ctx) {
 System.out.println("Handler 被添加到 pipeline");
 }

 @Override
 public void channelRegistered(ChannelHandlerContext ctx) {
 System.out.println("Channel 注册到 EventLoop");
 }

 @Override
 public void channelActive(ChannelHandlerContext ctx) {
 System.out.println("Channel 变为活跃状态，可以读写数据");

 // 发送欢迎消息
 String welcomeMsg = "欢迎连接到 Netty 服务器！\n";
 ctx.writeAndFlush(Unpooled.copiedBuffer(welcomeMsg, CharsetUtil.UTF_8));
 }

 @Override
 public void channelRead(ChannelHandlerContext ctx, Object msg) {
 ByteBuf buf = (ByteBuf) msg;
 try {
 String receivedData = buf.toString(CharsetUtil.UTF_8);
 System.out.println("接收到数据: " + receivedData);

 // 回显数据
 String response = "Echo: " + receivedData;
 ctx.writeAndFlush(Unpooled.copiedBuffer(response, CharsetUtil.UTF_8));

 } finally {
 buf.release(); // 重要：释放 ByteBuf
 }
 }

 @Override
 public void channelReadComplete(ChannelHandlerContext ctx) {
 System.out.println("Channel 读取完成");
 ctx.flush(); // 确保数据被发送
 }

 @Override
 public void channelInactive(ChannelHandlerContext ctx) {
 System.out.println("Channel 变为非活跃状态");
 }

 @Override
 public void channelUnregistered(ChannelHandlerContext ctx) {
 System.out.println("Channel 从 EventLoop 注销");
 }

 @Override
 public void handlerRemoved(ChannelHandlerContext ctx) {
 System.out.println("Handler 从 pipeline 移除");
 }

 @Override
 public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
 System.err.println("异常处理: " + cause.getMessage());
 cause.printStackTrace();
 ctx.close(); // 关闭连接
 }
}

ChannelPipeline 深入理解

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


// Pipeline 配置和管理
public class PipelineManager {

 public static void configurePipeline(SocketChannel ch) {
 ChannelPipeline pipeline = ch.pipeline();

 // 1. 日志处理器（最前面）
 pipeline.addFirst("logger", new LoggingHandler(LogLevel.INFO));

 // 2. 空闲检测
 pipeline.addLast("idleState", new IdleStateHandler(30, 0, 0, TimeUnit.SECONDS));

 // 3. 协议解码器
 pipeline.addLast("frameDecoder", new LengthFieldBasedFrameDecoder(1024 * 1024, 0, 4, 0, 4));
 pipeline.addLast("frameEncoder", new LengthFieldPrepender(4));

 // 4. 字符串编解码
 pipeline.addLast("stringDecoder", new StringDecoder(CharsetUtil.UTF_8));
 pipeline.addLast("stringEncoder", new StringEncoder(CharsetUtil.UTF_8));

 // 5. 业务处理器
 pipeline.addLast("businessHandler", new BusinessHandler());

 // 6. 异常处理器（最后面）
 pipeline.addLast("exceptionHandler", new ExceptionHandler());
 }

 // 动态添加处理器
 public static void addSSLHandler(ChannelPipeline pipeline, SslContext sslContext) {
 pipeline.addFirst("ssl", sslContext.newHandler(pipeline.channel().alloc()));
 }

 // 动态移除处理器
 public static void removeHandler(ChannelPipeline pipeline, String name) {
 if (pipeline.get(name) != null) {
 pipeline.remove(name);
 }
 }
}

// 空闲状态处理
class IdleHandler extends ChannelInboundHandlerAdapter {
 @Override
 public void userEventTriggered(ChannelHandlerContext ctx, Object evt) throws Exception {
 if (evt instanceof IdleStateEvent) {
 IdleStateEvent event = (IdleStateEvent) evt;
 if (event.state() == IdleState.READER_IDLE) {
 System.out.println("读空闲，关闭连接");
 ctx.close();
 }
 } else {
 super.userEventTriggered(ctx, evt);
 }
 }
}

2.3 ByteBuf 内存管理

ByteBuf 基础操作

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85


// ByteBuf 详细使用示例
public class ByteBufExample {

 public static void basicOperations() {
 // 创建 ByteBuf
 ByteBuf buffer = Unpooled.buffer(256);

 System.out.println("初始状态:");
 printBufferInfo(buffer);

 // 写入数据
 buffer.writeBytes("Hello".getBytes());
 buffer.writeInt(42);
 buffer.writeLong(System.currentTimeMillis());

 System.out.println("写入数据后:");
 printBufferInfo(buffer);

 // 读取数据
 byte[] helloBytes = new byte[5];
 buffer.readBytes(helloBytes);
 int number = buffer.readInt();
 long timestamp = buffer.readLong();

 System.out.println("读取的数据:");
 System.out.println("String: " + new String(helloBytes));
 System.out.println("Int: " + number);
 System.out.println("Long: " + timestamp);

 System.out.println("读取数据后:");
 printBufferInfo(buffer);

 // 重要：释放 ByteBuf
 buffer.release();
 }

 private static void printBufferInfo(ByteBuf buf) {
 System.out.println("readerIndex: " + buf.readerIndex());
 System.out.println("writerIndex: " + buf.writerIndex());
 System.out.println("capacity: " + buf.capacity());
 System.out.println("readableBytes: " + buf.readableBytes());
 System.out.println("writableBytes: " + buf.writableBytes());
 System.out.println("---");
 }

 // 内存池使用
 public static void pooledByteBuf() {
 ByteBufAllocator allocator = PooledByteBufAllocator.DEFAULT;

 // 申请堆内存
 ByteBuf heapBuf = allocator.heapBuffer(256);

 // 申请直接内存（堆外内存）
 ByteBuf directBuf = allocator.directBuffer(256);

 try {
 // 使用缓冲区
 heapBuf.writeBytes("Heap Buffer".getBytes());
 directBuf.writeBytes("Direct Buffer".getBytes());

 System.out.println("Heap Buffer: " + heapBuf.toString(CharsetUtil.UTF_8));
 System.out.println("Direct Buffer: " + directBuf.toString(CharsetUtil.UTF_8));

 } finally {
 // 释放内存
 heapBuf.release();
 directBuf.release();
 }
 }

 // 复合缓冲区
 public static void compositeByteBuf() {
 CompositeByteBuf composite = Unpooled.compositeBuffer();

 ByteBuf header = Unpooled.copiedBuffer("Header", CharsetUtil.UTF_8);
 ByteBuf body = Unpooled.copiedBuffer("Body Content", CharsetUtil.UTF_8);

 // 将多个 ByteBuf 组合
 composite.addComponents(true, header, body);

 System.out.println("Composite Buffer: " + composite.toString(CharsetUtil.UTF_8));

 composite.release();
 }
}

内存泄漏检测

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47


// 内存泄漏检测配置
public class MemoryLeakDetection {

 static {
 // 设置内存泄漏检测级别
 System.setProperty("io.netty.leakDetection.level", "ADVANCED");
 // 可选值：DISABLED, SIMPLE, ADVANCED, PARANOID
 }

 // 正确的 ByteBuf 使用模式
 public static void correctByteBufUsage(ChannelHandlerContext ctx, Object msg) {
 ByteBuf buf = (ByteBuf) msg;
 try {
 // 处理数据
 String data = buf.toString(CharsetUtil.UTF_8);
 System.out.println("Received: " + data);

 // 创建响应
 ByteBuf response = ctx.alloc().buffer();
 try {
 response.writeBytes("Response".getBytes());
 ctx.writeAndFlush(response);
 response = null; // 防止在 finally 中重复释放
 } finally {
 if (response != null) {
 response.release();
 }
 }

 } finally {
 buf.release(); // 确保释放输入缓冲区
 }
 }

 // 使用 ReferenceCountUtil 安全释放
 public static void safeByteBufUsage(Object msg) {
 try {
 if (msg instanceof ByteBuf) {
 ByteBuf buf = (ByteBuf) msg;
 String data = buf.toString(CharsetUtil.UTF_8);
 System.out.println("Data: " + data);
 }
 } finally {
 ReferenceCountUtil.release(msg); // 安全释放
 }
 }
}

三、实战入门案例

3.1 Echo 服务器

服务器端实现

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120


// 完整的 Echo 服务器实现
public class EchoServer {
 private final int port;

 public EchoServer(int port) {
 this.port = port;
 }

 public void start() throws InterruptedException {
 EventLoopGroup bossGroup = new NioEventLoopGroup(1);
 EventLoopGroup workerGroup = new NioEventLoopGroup();

 try {
 ServerBootstrap b = new ServerBootstrap();
 b.group(bossGroup, workerGroup)
 .channel(NioServerSocketChannel.class)
 .option(ChannelOption.SO_BACKLOG, 1024)
 .option(ChannelOption.SO_REUSEADDR, true)
 .childOption(ChannelOption.SO_KEEPALIVE, true)
 .childOption(ChannelOption.TCP_NODELAY, true)
 .childOption(ChannelOption.SO_RCVBUF, 32 * 1024)
 .childOption(ChannelOption.SO_SNDBUF, 32 * 1024)
 .handler(new LoggingHandler(LogLevel.INFO))
 .childHandler(new ChannelInitializer<SocketChannel>() {
 @Override
 public void initChannel(SocketChannel ch) {
 ChannelPipeline p = ch.pipeline();

 // 添加处理器
 p.addLast(new IdleStateHandler(60, 30, 0, TimeUnit.SECONDS));
 p.addLast(new EchoServerHandler());
 }
 });

 // 绑定端口并启动服务器
 ChannelFuture f = b.bind(port).sync();
 System.out.println("Echo Server started on port " + port);

 // 等待服务器 socket 关闭
 f.channel().closeFuture().sync();

 } finally {
 workerGroup.shutdownGracefully();
 bossGroup.shutdownGracefully();
 }
 }

 public static void main(String[] args) throws InterruptedException {
 int port = args.length > 0 ? Integer.parseInt(args[0]) : 8080;
 new EchoServer(port).start();
 }
}

// Echo 服务器处理器
class EchoServerHandler extends ChannelInboundHandlerAdapter {
 private static final AtomicLong connectionCount = new AtomicLong(0);
 private static final AtomicLong messageCount = new AtomicLong(0);

 @Override
 public void channelActive(ChannelHandlerContext ctx) {
 long connections = connectionCount.incrementAndGet();
 System.out.println("Client connected: " + ctx.channel().remoteAddress() +
 ", Total connections: " + connections);
 }

 @Override
 public void channelInactive(ChannelHandlerContext ctx) {
 long connections = connectionCount.decrementAndGet();
 System.out.println("Client disconnected: " + ctx.channel().remoteAddress() +
 ", Total connections: " + connections);
 }

 @Override
 public void channelRead(ChannelHandlerContext ctx, Object msg) {
 ByteBuf in = (ByteBuf) msg;
 try {
 String message = in.toString(CharsetUtil.UTF_8);
 long msgCount = messageCount.incrementAndGet();

 System.out.println("Received message #" + msgCount + ": " + message.trim());

 // 构建响应
 String response = "Echo #" + msgCount + ": " + message;
 ByteBuf responseBuffer = Unpooled.copiedBuffer(response, CharsetUtil.UTF_8);

 ctx.writeAndFlush(responseBuffer);

 } finally {
 in.release();
 }
 }

 @Override
 public void userEventTriggered(ChannelHandlerContext ctx, Object evt) throws Exception {
 if (evt instanceof IdleStateEvent) {
 IdleStateEvent event = (IdleStateEvent) evt;
 String eventType = null;
 switch (event.state()) {
 case READER_IDLE:
 eventType = "读空闲";
 break;
 case WRITER_IDLE:
 eventType = "写空闲";
 break;
 case ALL_IDLE:
 eventType = "读写空闲";
 break;
 }
 System.out.println(ctx.channel().remoteAddress() + " 超时事件: " + eventType);
 ctx.channel().close();
 }
 }

 @Override
 public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
 System.err.println("Server exception: " + cause.getMessage());
 cause.printStackTrace();
 ctx.close();
 }
}

客户端实现

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95


// Echo 客户端实现
public class EchoClient {
 private final String host;
 private final int port;

 public EchoClient(String host, int port) {
 this.host = host;
 this.port = port;
 }

 public void start() throws InterruptedException {
 EventLoopGroup group = new NioEventLoopGroup();

 try {
 Bootstrap b = new Bootstrap();
 b.group(group)
 .channel(NioSocketChannel.class)
 .option(ChannelOption.TCP_NODELAY, true)
 .option(ChannelOption.SO_KEEPALIVE, true)
 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 5000)
 .handler(new ChannelInitializer<SocketChannel>() {
 @Override
 public void initChannel(SocketChannel ch) {
 ChannelPipeline p = ch.pipeline();
 p.addLast(new EchoClientHandler());
 }
 });

 // 连接服务器
 ChannelFuture f = b.connect(host, port).sync();
 System.out.println("Connected to " + host + ":" + port);

 // 等待连接关闭
 f.channel().closeFuture().sync();

 } finally {
 group.shutdownGracefully();
 }
 }

 public static void main(String[] args) throws InterruptedException {
 String host = args.length > 0 ? args[0] : "localhost";
 int port = args.length > 1 ? Integer.parseInt(args[1]) : 8080;

 new EchoClient(host, port).start();
 }
}

// Echo 客户端处理器
class EchoClientHandler extends ChannelInboundHandlerAdapter {
 private final AtomicInteger messageCounter = new AtomicInteger(0);

 @Override
 public void channelActive(ChannelHandlerContext ctx) {
 System.out.println("Connected to server: " + ctx.channel().remoteAddress());

 // 开始发送消息
 sendMessage(ctx);
 }

 private void sendMessage(ChannelHandlerContext ctx) {
 // 定时发送消息
 ctx.executor().schedule(() -> {
 if (ctx.channel().isActive()) {
 int msgNum = messageCounter.incrementAndGet();
 String message = "Hello Netty! Message #" + msgNum +
 " at " + new Date() + "\n";

 ByteBuf buffer = Unpooled.copiedBuffer(message, CharsetUtil.UTF_8);
 ctx.writeAndFlush(buffer);

 // 继续发送下一条消息
 sendMessage(ctx);
 }
 }, 2, TimeUnit.SECONDS);
 }

 @Override
 public void channelRead(ChannelHandlerContext ctx, Object msg) {
 ByteBuf in = (ByteBuf) msg;
 try {
 String response = in.toString(CharsetUtil.UTF_8);
 System.out.println("Server response: " + response.trim());
 } finally {
 in.release();
 }
 }

 @Override
 public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
 System.err.println("Client exception: " + cause.getMessage());
 cause.printStackTrace();
 ctx.close();
 }
}

3.2 HTTP 服务器

完整的 HTTP 服务器

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150


// HTTP 服务器实现
public class HttpServer {
 private final int port;

 public HttpServer(int port) {
 this.port = port;
 }

 public void start() throws InterruptedException {
 EventLoopGroup bossGroup = new NioEventLoopGroup(1);
 EventLoopGroup workerGroup = new NioEventLoopGroup();

 try {
 ServerBootstrap b = new ServerBootstrap();
 b.group(bossGroup, workerGroup)
 .channel(NioServerSocketChannel.class)
 .option(ChannelOption.SO_BACKLOG, 1024)
 .childOption(ChannelOption.SO_KEEPALIVE, true)
 .childOption(ChannelOption.TCP_NODELAY, true)
 .childHandler(new HttpServerInitializer());

 ChannelFuture f = b.bind(port).sync();
 System.out.println("HTTP Server started on http://localhost:" + port);

 f.channel().closeFuture().sync();

 } finally {
 workerGroup.shutdownGracefully();
 bossGroup.shutdownGracefully();
 }
 }

 public static void main(String[] args) throws InterruptedException {
 int port = args.length > 0 ? Integer.parseInt(args[0]) : 8080;
 new HttpServer(port).start();
 }
}

// HTTP 服务器初始化器
class HttpServerInitializer extends ChannelInitializer<SocketChannel> {
 @Override
 public void initChannel(SocketChannel ch) {
 ChannelPipeline pipeline = ch.pipeline();

 // HTTP 编解码器
 pipeline.addLast(new HttpServerCodec());

 // HTTP 对象聚合器
 pipeline.addLast(new HttpObjectAggregator(65536));

 // HTTP 内容压缩
 pipeline.addLast(new HttpContentCompressor());

 // 自定义业务处理器
 pipeline.addLast(new HttpServerHandler());
 }
}

// HTTP 服务器处理器
class HttpServerHandler extends ChannelInboundHandlerAdapter {
 private static final Map<String, String> routes = new HashMap<>();
 private static final AtomicLong requestCount = new AtomicLong(0);

 static {
 // 配置路由
 routes.put("/", "Welcome to Netty HTTP Server!");
 routes.put("/hello", "Hello, World!");
 routes.put("/time", "Current time: " + new Date());
 routes.put("/stats", "Request count: ");
 }

 @Override
 public void channelRead(ChannelHandlerContext ctx, Object msg) {
 if (msg instanceof FullHttpRequest) {
 FullHttpRequest request = (FullHttpRequest) msg;

 long count = requestCount.incrementAndGet();
 String uri = request.uri();
 HttpMethod method = request.method();

 System.out.println("Request #" + count + ": " + method + " " + uri);

 // 处理请求
 handleHttpRequest(ctx, request, count);
 }
 }

 private void handleHttpRequest(ChannelHandlerContext ctx, FullHttpRequest request, long count) {
 String uri = request.uri();
 String responseContent;
 HttpResponseStatus status = HttpResponseStatus.OK;

 // 路由处理
 if (routes.containsKey(uri)) {
 responseContent = routes.get(uri);
 if (uri.equals("/stats")) {
 responseContent += count;
 }
 } else if (uri.equals("/api/echo")) {
 // API 接口：回显请求体
 ByteBuf content = request.content();
 responseContent = "Echo: " + content.toString(CharsetUtil.UTF_8);
 } else if (uri.startsWith("/api/user/")) {
 // RESTful API 示例
 String userId = uri.substring("/api/user/".length());
 responseContent = "{\"userId\": \"" + userId + "\", \"name\": \"User " + userId + "\"}";
 } else {
 // 404 处理
 responseContent = "404 Not Found: " + uri;
 status = HttpResponseStatus.NOT_FOUND;
 }

 // 构建 HTTP 响应
 sendHttpResponse(ctx, request, responseContent, status);
 }

 private void sendHttpResponse(ChannelHandlerContext ctx, FullHttpRequest request,
 String content, HttpResponseStatus status) {

 ByteBuf responseContent = Unpooled.copiedBuffer(content, CharsetUtil.UTF_8);

 FullHttpResponse response = new DefaultFullHttpResponse(
 HttpVersion.HTTP_1_1, status, responseContent);

 // 设置响应头
 response.headers().set(HttpHeaderNames.CONTENT_TYPE, "text/plain; charset=UTF-8");
 response.headers().set(HttpHeaderNames.CONTENT_LENGTH, responseContent.readableBytes());
 response.headers().set(HttpHeaderNames.SERVER, "Netty/4.1");
 response.headers().set(HttpHeaderNames.DATE, new Date());

 // 处理 Keep-Alive
 boolean keepAlive = HttpUtil.isKeepAlive(request);
 if (keepAlive) {
 response.headers().set(HttpHeaderNames.CONNECTION, HttpHeaderValues.KEEP_ALIVE);
 }

 // 发送响应
 ChannelFuture future = ctx.writeAndFlush(response);

 if (!keepAlive) {
 future.addListener(ChannelFutureListener.CLOSE);
 }
 }

 @Override
 public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
 System.err.println("HTTP Server exception: " + cause.getMessage());
 ctx.close();
 }
}

3.3 WebSocket 服务器

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108


// WebSocket 服务器实现
public class WebSocketServer {
 private final int port;

 public WebSocketServer(int port) {
 this.port = port;
 }

 public void start() throws InterruptedException {
 EventLoopGroup bossGroup = new NioEventLoopGroup(1);
 EventLoopGroup workerGroup = new NioEventLoopGroup();

 try {
 ServerBootstrap b = new ServerBootstrap();
 b.group(bossGroup, workerGroup)
 .channel(NioServerSocketChannel.class)
 .childHandler(new WebSocketServerInitializer());

 ChannelFuture f = b.bind(port).sync();
 System.out.println("WebSocket Server started on ws://localhost:" + port + "/websocket");

 f.channel().closeFuture().sync();

 } finally {
 workerGroup.shutdownGracefully();
 bossGroup.shutdownGracefully();
 }
 }

 public static void main(String[] args) throws InterruptedException {
 int port = args.length > 0 ? Integer.parseInt(args[0]) : 8080;
 new WebSocketServer(port).start();
 }
}

// WebSocket 服务器初始化器
class WebSocketServerInitializer extends ChannelInitializer<SocketChannel> {
 @Override
 public void initChannel(SocketChannel ch) {
 ChannelPipeline pipeline = ch.pipeline();

 pipeline.addLast(new HttpServerCodec());
 pipeline.addLast(new HttpObjectAggregator(65536));
 pipeline.addLast(new WebSocketServerProtocolHandler("/websocket", null, true));
 pipeline.addLast(new WebSocketServerHandler());
 }
}

// WebSocket 处理器
class WebSocketServerHandler extends SimpleChannelInboundHandler<WebSocketFrame> {
 private static final Set<Channel> channels = ConcurrentHashMap.newKeySet();

 @Override
 public void channelActive(ChannelHandlerContext ctx) {
 channels.add(ctx.channel());
 System.out.println("WebSocket client connected: " + ctx.channel().remoteAddress());
 System.out.println("Total clients: " + channels.size());
 }

 @Override
 public void channelInactive(ChannelHandlerContext ctx) {
 channels.remove(ctx.channel());
 System.out.println("WebSocket client disconnected: " + ctx.channel().remoteAddress());
 System.out.println("Total clients: " + channels.size());
 }

 @Override
 protected void channelRead0(ChannelHandlerContext ctx, WebSocketFrame frame) {
 if (frame instanceof TextWebSocketFrame) {
 // 处理文本消息
 String text = ((TextWebSocketFrame) frame).text();
 System.out.println("Received: " + text);

 // 广播消息给所有客户端
 String response = "Server Echo: " + text + " (Clients: " + channels.size() + ")";
 TextWebSocketFrame responseFrame = new TextWebSocketFrame(response);

 // 广播给所有连接的客户端
 for (Channel channel : channels) {
 if (channel.isActive()) {
 channel.writeAndFlush(responseFrame.retainedDuplicate());
 }
 }

 } else if (frame instanceof BinaryWebSocketFrame) {
 // 处理二进制消息
 ByteBuf content = frame.content();
 System.out.println("Received binary data: " + content.readableBytes() + " bytes");

 // 回显二进制数据
 ctx.writeAndFlush(new BinaryWebSocketFrame(content.retain()));

 } else if (frame instanceof PingWebSocketFrame) {
 // 处理 Ping 帧
 ctx.writeAndFlush(new PongWebSocketFrame(frame.content().retain()));

 } else if (frame instanceof CloseWebSocketFrame) {
 // 处理关闭帧
 ctx.close();
 }
 }

 @Override
 public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
 System.err.println("WebSocket exception: " + cause.getMessage());
 ctx.close();
 }
}

四、性能分析与优化

4.1 性能基准测试

让我们看看 Netty 在不同场景下的性能表现：

从图中可以看出，Netty 的 NIO 模型在 CPU 使用率和吞吐量之间取得了很好的平衡。

在内存使用方面，Netty 表现出色，即使在高并发场景下也能保持较低的内存占用。

Netty 在处理并发连接方面表现优异，能够快速建立大量连接并保持稳定。

4.2 性能优化技巧

JVM 优化

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# Netty 服务器 JVM 优化参数
java -server \
 -Xms4g -Xmx4g \
 -XX:+UseG1GC \
 -XX:MaxGCPauseMillis=200 \
 -XX:+UnlockExperimentalVMOptions \
 -XX:+UseZGC \
 -XX:+DisableExplicitGC \
 -XX:+HeapDumpOnOutOfMemoryError \
 -XX:HeapDumpPath=/tmp/netty-heapdump.hprof \
 -Dio.netty.leakDetection.level=SIMPLE \
 -Dio.netty.allocator.type=pooled \
 -Dio.netty.allocator.numDirectArenas=32 \
 -Dio.netty.allocator.numHeapArenas=32 \
 -Dio.netty.recycler.maxCapacityPerThread=0 \
 -cp your-app.jar com.example.NettyServer

代码优化

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81


// Netty 性能优化配置
public class NettyPerformanceOptimization {

 public static ServerBootstrap createOptimizedServer() {
 // 创建优化的 EventLoopGroup
 EventLoopGroup bossGroup = new NioEventLoopGroup(1,
 new DefaultThreadFactory("NettyBoss", true));
 EventLoopGroup workerGroup = new NioEventLoopGroup(
 Runtime.getRuntime().availableProcessors() * 2,
 new DefaultThreadFactory("NettyWorker", true));

 ServerBootstrap bootstrap = new ServerBootstrap();
 bootstrap.group(bossGroup, workerGroup)
 .channel(NioServerSocketChannel.class)
 // 网络层优化
 .option(ChannelOption.SO_BACKLOG, 1024)
 .option(ChannelOption.SO_REUSEADDR, true)
 .childOption(ChannelOption.SO_KEEPALIVE, true)
 .childOption(ChannelOption.TCP_NODELAY, true)
 .childOption(ChannelOption.SO_RCVBUF, 32 * 1024)
 .childOption(ChannelOption.SO_SNDBUF, 32 * 1024)
 // 内存优化
 .childOption(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT)
 .childOption(ChannelOption.RCVBUF_ALLOCATOR,
 new AdaptiveRecvByteBufAllocator(64, 1024, 65536))
 // 写缓冲区优化
 .childOption(ChannelOption.WRITE_BUFFER_WATER_MARK,
 new WriteBufferWaterMark(8 * 1024, 32 * 1024));

 return bootstrap;
 }

 // 高性能编解码器
 public static class OptimizedStringDecoder extends MessageToMessageDecoder<ByteBuf> {
 private final Charset charset;

 public OptimizedStringDecoder(Charset charset) {
 this.charset = charset;
 }

 @Override
 protected void decode(ChannelHandlerContext ctx, ByteBuf msg, List<Object> out) {
 // 零拷贝字符串解码
 out.add(msg.toString(charset));
 }
 }

 // 批量写优化
 public static class BatchedWriteHandler extends ChannelOutboundHandlerAdapter {
 private final List<Object> pendingWrites = new ArrayList<>();
 private final int batchSize;

 public BatchedWriteHandler(int batchSize) {
 this.batchSize = batchSize;
 }

 @Override
 public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise) {
 pendingWrites.add(msg);

 if (pendingWrites.size() >= batchSize) {
 flushBatch(ctx);
 }
 }

 @Override
 public void flush(ChannelHandlerContext ctx) {
 if (!pendingWrites.isEmpty()) {
 flushBatch(ctx);
 }
 ctx.flush();
 }

 private void flushBatch(ChannelHandlerContext ctx) {
 for (Object msg : pendingWrites) {
 ctx.write(msg);
 }
 pendingWrites.clear();
 }
 }
}

操作系统优化

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# Linux 系统优化配置
echo 'net.core.rmem_max = 134217728' >> /etc/sysctl.conf
echo 'net.core.wmem_max = 134217728' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_rmem = 4096 16384 134217728' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_wmem = 4096 16384 134217728' >> /etc/sysctl.conf
echo 'net.core.netdev_max_backlog = 5000' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_congestion_control = bbr' >> /etc/sysctl.conf

# 文件描述符限制
echo '* soft nofile 1000000' >> /etc/security/limits.conf
echo '* hard nofile 1000000' >> /etc/security/limits.conf

# 应用配置
sysctl -p

4.3 监控和调试

性能监控

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56


// Netty 性能监控
public class NettyMonitor {
 private static final MeterRegistry registry = Metrics.globalRegistry;
 private static final Timer requestTimer = Timer.builder("netty.request.duration")
 .register(registry);
 private static final Counter connectionCounter = Counter.builder("netty.connections.total")
 .register(registry);

 // 监控处理器
 public static class MonitoringHandler extends ChannelInboundHandlerAdapter {
 @Override
 public void channelActive(ChannelHandlerContext ctx) {
 connectionCounter.increment();
 super.channelActive(ctx);
 }

 @Override
 public void channelRead(ChannelHandlerContext ctx, Object msg) {
 Timer.Sample sample = Timer.start(registry);

 try {
 super.channelRead(ctx, msg);
 } finally {
 sample.stop(requestTimer);
 }
 }

 @Override
 public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
 Counter.builder("netty.errors.total")
 .tag("error.type", cause.getClass().getSimpleName())
 .register(registry)
 .increment();

 super.exceptionCaught(ctx, cause);
 }
 }

 // 定期输出监控指标
 public static void startMetricsReporting() {
 ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);

 scheduler.scheduleAtFixedRate(() -> {
 System.out.println("=== Netty Metrics ===");
 System.out.println("Total connections: " + connectionCounter.count());
 System.out.println("Request rate: " + requestTimer.count() + " req/s");
 System.out.println("Average latency: " +
 requestTimer.mean(TimeUnit.MILLISECONDS) + " ms");
 System.out.println("P95 latency: " +
 requestTimer.percentile(0.95, TimeUnit.MILLISECONDS) + " ms");
 System.out.println("P99 latency: " +
 requestTimer.percentile(0.99, TimeUnit.MILLISECONDS) + " ms");

 }, 0, 30, TimeUnit.SECONDS);
 }
}

五、高级特性

5.1 自定义协议

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121


// 自定义协议实现
public class CustomProtocol {

 // 协议格式：[4字节长度][1字节类型][数据]
 public static class CustomMessage {
 private final byte type;
 private final byte[] data;

 public CustomMessage(byte type, byte[] data) {
 this.type = type;
 this.data = data;
 }

 // getter 方法
 public byte getType() { return type; }
 public byte[] getData() { return data; }
 public int getLength() { return 1 + (data != null ? data.length : 0); }
 }

 // 自定义协议编码器
 public static class CustomProtocolEncoder extends MessageToByteEncoder<CustomMessage> {
 @Override
 protected void encode(ChannelHandlerContext ctx, CustomMessage msg, ByteBuf out) {
 // 写入长度（4字节）
 out.writeInt(msg.getLength());
 // 写入类型（1字节）
 out.writeByte(msg.getType());
 // 写入数据
 if (msg.getData() != null) {
 out.writeBytes(msg.getData());
 }
 }
 }

 // 自定义协议解码器
 public static class CustomProtocolDecoder extends LengthFieldBasedFrameDecoder {
 public CustomProtocolDecoder() {
 super(1024 * 1024, 0, 4, 0, 4);
 }

 @Override
 protected Object decode(ChannelHandlerContext ctx, ByteBuf in) throws Exception {
 ByteBuf frame = (ByteBuf) super.decode(ctx, in);
 if (frame == null) {
 return null;
 }

 try {
 byte type = frame.readByte();
 byte[] data = new byte[frame.readableBytes()];
 frame.readBytes(data);

 return new CustomMessage(type, data);
 } finally {
 frame.release();
 }
 }
 }

 // 协议处理器
 public static class CustomProtocolHandler extends SimpleChannelInboundHandler<CustomMessage> {
 @Override
 protected void channelRead0(ChannelHandlerContext ctx, CustomMessage msg) {
 byte type = msg.getType();
 byte[] data = msg.getData();

 switch (type) {
 case 0x01: // 心跳
 handleHeartbeat(ctx);
 break;
 case 0x02: // 数据传输
 handleDataTransfer(ctx, data);
 break;
 case 0x03: // 控制命令
 handleControlCommand(ctx, data);
 break;
 default:
 System.err.println("Unknown message type: " + type);
 }
 }

 private void handleHeartbeat(ChannelHandlerContext ctx) {
 // 回复心跳
 CustomMessage pong = new CustomMessage((byte) 0x01, "PONG".getBytes());
 ctx.writeAndFlush(pong);
 }

 private void handleDataTransfer(ChannelHandlerContext ctx, byte[] data) {
 String dataStr = new String(data, CharsetUtil.UTF_8);
 System.out.println("Received data: " + dataStr);

 // 回复确认
 CustomMessage ack = new CustomMessage((byte) 0x04, "ACK".getBytes());
 ctx.writeAndFlush(ack);
 }

 private void handleControlCommand(ChannelHandlerContext ctx, byte[] data) {
 String command = new String(data, CharsetUtil.UTF_8);
 System.out.println("Received command: " + command);

 // 执行命令并回复结果
 String result = executeCommand(command);
 CustomMessage response = new CustomMessage((byte) 0x05, result.getBytes());
 ctx.writeAndFlush(response);
 }

 private String executeCommand(String command) {
 // 简单的命令处理示例
 switch (command.toLowerCase()) {
 case "status":
 return "Server is running";
 case "time":
 return new Date().toString();
 case "shutdown":
 return "Shutdown initiated";
 default:
 return "Unknown command: " + command;
 }
 }
 }
}

5.2 SSL/TLS 支持

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62


// SSL/TLS 配置
public class SSLNettyServer {

 public static SslContext createSSLContext() throws Exception {
 // 创建自签名证书（仅用于测试）
 SelfSignedCertificate ssc = new SelfSignedCertificate();

 return SslContextBuilder.forServer(ssc.certificate(), ssc.privateKey())
 .sslProvider(SslProvider.OPENSSL) // 使用 OpenSSL（性能更好）
 .ciphers(Http2SecurityUtil.CIPHERS, SupportedCipherSuiteFilter.INSTANCE)
 .applicationProtocolConfig(new ApplicationProtocolConfig(
 ApplicationProtocolConfig.Protocol.ALPN,
 ApplicationProtocolConfig.SelectorFailureBehavior.NO_ADVERTISE,
 ApplicationProtocolConfig.SelectedListenerFailureBehavior.ACCEPT,
 ApplicationProtocolNames.HTTP_2,
 ApplicationProtocolNames.HTTP_1_1))
 .build();
 }

 // SSL 初始化器
 public static class SSLChannelInitializer extends ChannelInitializer<SocketChannel> {
 private final SslContext sslContext;

 public SSLChannelInitializer(SslContext sslContext) {
 this.sslContext = sslContext;
 }

 @Override
 protected void initChannel(SocketChannel ch) throws Exception {
 ChannelPipeline pipeline = ch.pipeline();

 // 添加 SSL 处理器（必须是第一个）
 pipeline.addLast(sslContext.newHandler(ch.alloc()));

 // 添加其他处理器
 pipeline.addLast(new HttpServerCodec());
 pipeline.addLast(new HttpObjectAggregator(65536));
 pipeline.addLast(new HttpsServerHandler());
 }
 }

 // HTTPS 处理器
 public static class HttpsServerHandler extends SimpleChannelInboundHandler<FullHttpRequest> {
 @Override
 protected void channelRead0(ChannelHandlerContext ctx, FullHttpRequest request) {
 String uri = request.uri();

 // 构建 HTTPS 响应
 String responseContent = "HTTPS Response for: " + uri + "\nSecure: true\n";
 ByteBuf content = Unpooled.copiedBuffer(responseContent, CharsetUtil.UTF_8);

 FullHttpResponse response = new DefaultFullHttpResponse(
 HttpVersion.HTTP_1_1, HttpResponseStatus.OK, content);

 response.headers().set(HttpHeaderNames.CONTENT_TYPE, "text/plain");
 response.headers().set(HttpHeaderNames.CONTENT_LENGTH, content.readableBytes());
 response.headers().set("Strict-Transport-Security", "max-age=31536000");

 ctx.writeAndFlush(response).addListener(ChannelFutureListener.CLOSE);
 }
 }
}

5.3 HTTP/2 支持

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116


// HTTP/2 服务器实现
public class Http2Server {

 public static void main(String[] args) throws Exception {
 SslContext sslCtx = createHttp2SSLContext();

 EventLoopGroup bossGroup = new NioEventLoopGroup(1);
 EventLoopGroup workerGroup = new NioEventLoopGroup();

 try {
 ServerBootstrap b = new ServerBootstrap();
 b.group(bossGroup, workerGroup)
 .channel(NioServerSocketChannel.class)
 .childHandler(new Http2ServerInitializer(sslCtx));

 ChannelFuture f = b.bind(8443).sync();
 System.out.println("HTTP/2 Server started on https://localhost:8443");

 f.channel().closeFuture().sync();

 } finally {
 workerGroup.shutdownGracefully();
 bossGroup.shutdownGracefully();
 }
 }

 private static SslContext createHttp2SSLContext() throws Exception {
 SelfSignedCertificate ssc = new SelfSignedCertificate();

 return SslContextBuilder.forServer(ssc.certificate(), ssc.privateKey())
 .sslProvider(SslProvider.OPENSSL)
 .ciphers(Http2SecurityUtil.CIPHERS, SupportedCipherSuiteFilter.INSTANCE)
 .applicationProtocolConfig(new ApplicationProtocolConfig(
 ApplicationProtocolConfig.Protocol.ALPN,
 ApplicationProtocolConfig.SelectorFailureBehavior.NO_ADVERTISE,
 ApplicationProtocolConfig.SelectedListenerFailureBehavior.ACCEPT,
 ApplicationProtocolNames.HTTP_2))
 .build();
 }
}

// HTTP/2 初始化器
class Http2ServerInitializer extends ChannelInitializer<SocketChannel> {
 private final SslContext sslCtx;

 public Http2ServerInitializer(SslContext sslCtx) {
 this.sslCtx = sslCtx;
 }

 @Override
 protected void initChannel(SocketChannel ch) {
 ChannelPipeline pipeline = ch.pipeline();
 pipeline.addLast(sslCtx.newHandler(ch.alloc()));
 pipeline.addLast(new ApplicationProtocolNegotiationHandler("") {
 @Override
 protected void configurePipeline(ChannelHandlerContext ctx, String protocol) {
 if (ApplicationProtocolNames.HTTP_2.equals(protocol)) {
 configureHttp2(ctx);
 } else if (ApplicationProtocolNames.HTTP_1_1.equals(protocol)) {
 configureHttp1(ctx);
 } else {
 throw new IllegalStateException("Unknown protocol: " + protocol);
 }
 }
 });
 }

 private void configureHttp2(ChannelHandlerContext ctx) {
 // HTTP/2 配置
 Http2ConnectionHandler connectionHandler = new Http2ConnectionHandlerBuilder()
 .frameListener(new Http2FrameAdapter() {
 @Override
 public void onDataRead(ChannelHandlerContext ctx, int streamId,
 ByteBuf data, int padding, boolean endOfStream) {
 // 处理 HTTP/2 数据帧
 }

 @Override
 public void onHeadersRead(ChannelHandlerContext ctx, int streamId,
 Http2Headers headers, int padding, boolean endOfStream) {
 // 处理 HTTP/2 头部帧
 sendHttp2Response(ctx, streamId, headers);
 }
 })
 .build();

 ctx.pipeline().addLast(connectionHandler);
 }

 private void configureHttp1(ChannelHandlerContext ctx) {
 // HTTP/1.1 降级配置
 ChannelPipeline pipeline = ctx.pipeline();
 pipeline.addLast(new HttpServerCodec());
 pipeline.addLast(new HttpObjectAggregator(65536));
 pipeline.addLast(new HttpServerHandler());
 }

 private void sendHttp2Response(ChannelHandlerContext ctx, int streamId, Http2Headers headers) {
 // 构建 HTTP/2 响应
 String content = "HTTP/2 Response for stream " + streamId;
 ByteBuf responseContent = Unpooled.copiedBuffer(content, CharsetUtil.UTF_8);

 Http2Headers responseHeaders = new DefaultHttp2Headers()
 .status("200")
 .set("content-type", "text/plain")
 .setInt("content-length", responseContent.readableBytes());

 // 发送响应头
 ctx.writeAndFlush(new DefaultHttp2HeadersFrame(responseHeaders, false).stream(
 Http2CodecUtil.streamIdToInt(streamId)));

 // 发送响应数据
 ctx.writeAndFlush(new DefaultHttp2DataFrame(responseContent, true).stream(
 Http2CodecUtil.streamIdToInt(streamId)));
 }
}

六、最佳实践

6.1 生产环境配置

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


// 生产环境 Netty 服务器配置
public class ProductionNettyServer {

 public static ServerBootstrap createProductionServer() {
 // 优化的线程组配置
 EventLoopGroup bossGroup = new NioEventLoopGroup(1);
 EventLoopGroup workerGroup = new NioEventLoopGroup(
 Math.min(Runtime.getRuntime().availableProcessors() * 2, 32));

 ServerBootstrap bootstrap = new ServerBootstrap();
 bootstrap.group(bossGroup, workerGroup)
 .channel(NioServerSocketChannel.class)
 // Socket 选项优化
 .option(ChannelOption.SO_BACKLOG, 1024)
 .option(ChannelOption.SO_REUSEADDR, true)
 .childOption(ChannelOption.SO_KEEPALIVE, true)
 .childOption(ChannelOption.TCP_NODELAY, true)
 .childOption(ChannelOption.SO_RCVBUF, 32 * 1024)
 .childOption(ChannelOption.SO_SNDBUF, 32 * 1024)
 // 内存分配器优化
 .childOption(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT)
 .childOption(ChannelOption.RCVBUF_ALLOCATOR,
 new AdaptiveRecvByteBufAllocator(64, 1024, 65536))
 // 写缓冲区优化
 .childOption(ChannelOption.WRITE_BUFFER_WATER_MARK,
 new WriteBufferWaterMark(8 * 1024, 32 * 1024))
 // 连接超时
 .childOption(ChannelOption.CONNECT_TIMEOUT_MILLIS, 10000);

 return bootstrap;
 }

 // 优雅关闭
 public static void gracefulShutdown(EventLoopGroup... groups) {
 for (EventLoopGroup group : groups) {
 try {
 group.shutdownGracefully(5, 15, TimeUnit.SECONDS).sync();
 } catch (InterruptedException e) {
 Thread.currentThread().interrupt();
 }
 }
 }
}

6.2 错误处理

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


// 全局异常处理器
public class GlobalExceptionHandler extends ChannelInboundHandlerAdapter {
 private static final Logger logger = LoggerFactory.getLogger(GlobalExceptionHandler.class);

 @Override
 public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
 if (cause instanceof IOException) {
 logger.debug("Connection reset by peer: {}", ctx.channel().remoteAddress());
 } else if (cause instanceof TooLongFrameException) {
 logger.warn("Frame too long from: {}", ctx.channel().remoteAddress());
 sendErrorResponse(ctx, HttpResponseStatus.REQUEST_ENTITY_TOO_LARGE);
 } else if (cause instanceof DecoderException) {
 logger.warn("Decode error from: {}, error: {}",
 ctx.channel().remoteAddress(), cause.getMessage());
 sendErrorResponse(ctx, HttpResponseStatus.BAD_REQUEST);
 } else {
 logger.error("Unexpected exception from: {}",
 ctx.channel().remoteAddress(), cause);
 sendErrorResponse(ctx, HttpResponseStatus.INTERNAL_SERVER_ERROR);
 }

 ctx.close();
 }

 private void sendErrorResponse(ChannelHandlerContext ctx, HttpResponseStatus status) {
 String content = "Error: " + status.toString();
 ByteBuf buf = Unpooled.copiedBuffer(content, CharsetUtil.UTF_8);

 FullHttpResponse response = new DefaultFullHttpResponse(
 HttpVersion.HTTP_1_1, status, buf);

 response.headers().set(HttpHeaderNames.CONTENT_TYPE, "text/plain");
 response.headers().set(HttpHeaderNames.CONTENT_LENGTH, buf.readableBytes());

 ctx.writeAndFlush(response).addListener(ChannelFutureListener.CLOSE);
 }
}

6.3 日志和监控

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42


// 日志配置
public class NettyLogging {

 // 自定义日志处理器
 public static class CustomLoggingHandler extends LoggingHandler {
 public CustomLoggingHandler() {
 super("NettyServer", LogLevel.INFO);
 }

 @Override
 public void channelActive(ChannelHandlerContext ctx) throws Exception {
 logger.info("Client connected from: {}", ctx.channel().remoteAddress());
 super.channelActive(ctx);
 }

 @Override
 public void channelInactive(ChannelHandlerContext ctx) throws Exception {
 logger.info("Client disconnected: {}", ctx.channel().remoteAddress());
 super.channelInactive(ctx);
 }
 }

 // 请求日志处理器
 public static class RequestLoggingHandler extends ChannelInboundHandlerAdapter {
 private long startTime;

 @Override
 public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
 startTime = System.currentTimeMillis();
 super.channelRead(ctx, msg);
 }

 @Override
 public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise) throws Exception {
 promise.addListener(future -> {
 long duration = System.currentTimeMillis() - startTime;
 logger.info("Request processed in {}ms", duration);
 });
 super.write(ctx, msg, promise);
 }
 }
}

七、总结

Netty 作为业界领先的异步网络编程框架，在构建高性能网络应用方面具有无可比拟的优势。通过本文的深入讲解，你应该已经掌握了：

核心收获

理论基础：EventLoop、Channel、Pipeline 等核心概念
实战技能：HTTP 服务器、WebSocket、自定义协议实现
性能优化：JVM 调优、系统优化、代码优化技巧
高级特性：SSL/TLS、HTTP/2、自定义协议
最佳实践：生产环境配置、错误处理、监控日志

性能优势总结

从我们的性能测试图表中可以看出：

吞吐量：Netty 在各种负载下都表现出色，特别是小数据包场景
延迟：P99 延迟控制在 3ms 以内，远优于传统框架
并发：单机可支持数万并发连接，内存占用合理
资源利用：CPU 和内存使用效率高，扩展性好

应用场景

Netty 适用于以下场景：

微服务架构：服务间高性能通信
游戏服务器：实时通信和状态同步
消息中间件：高吞吐量消息处理
Web 服务器：高并发 HTTP 服务
IoT 平台：设备连接和数据收集

掌握 Netty，就掌握了构建高性能网络应用的核心技能。随着微服务和云原生技术的发展，Netty 的重要性将会越来越突出。

延伸学习：建议深入研究 Netty 源码，理解其内部实现机制，这对于编写高质量的网络应用代码大有裨益。

]]>

brpc media-server 完全指南：构建高性能流媒体服务

lesshash@blog.lesshash.com/ (lesshash) — Thu, 18 Sep 2025 15:30:00 +0800

一、brpc media-server 简介

1.1 项目概述

brpc media-server 是百度云开源的高性能流媒体服务器，基于 brpc 框架构建。它为构建可扩展的直播流媒体平台提供了完整的解决方案。

1.2 核心特性

协议支持

RTMP 协议：支持 RTMP 推流和拉流
HTTP-FLV：提供低延迟的 HTTP-FLV 流
HLS 协议：支持标准 HLS 和低延迟 HLS
HTTPS 支持：全面支持加密传输

架构优势

源站模式（Origin Server）：处理推流和播放请求
边缘模式（Edge Server）：作为代理服务器分发内容
高性能架构：基于 brpc 的高并发处理能力
灵活配置：丰富的配置选项满足不同场景需求

功能特性

流标识系统：使用 vhost/app/stream_name 格式标识流
缓冲队列：可配置的帧队列缓冲
重试策略：可配置的重试机制
监控接口：基于 HTTP 的状态监控
音视频分离：支持纯音频或纯视频流

二、环境搭建

2.1 系统要求

1
2
3
4
5
6
7
8


# 支持的系统
- Linux (推荐 Ubuntu 18.04+/CentOS 7+)
- macOS (用于开发测试)

# 依赖
- GCC 4.8+ 或 Clang 3.5+
- CMake 3.10+
- Git

2.2 依赖安装

Ubuntu/Debian 系统

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


# 更新包管理器
sudo apt-get update

# 安装基础依赖
sudo apt-get install -y \
 build-essential \
 cmake \
 git \
 pkg-config \
 libssl-dev \
 libgflags-dev \
 libprotobuf-dev \
 libprotoc-dev \
 protobuf-compiler \
 libleveldb-dev \
 libsnappy-dev \
 libgoogle-glog-dev

# 安装 FFmpeg (用于测试)
sudo apt-get install -y ffmpeg

CentOS/RHEL 系统

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# 安装 EPEL 源
sudo yum install -y epel-release

# 安装基础依赖
sudo yum groupinstall -y "Development Tools"
sudo yum install -y \
 cmake3 \
 git \
 openssl-devel \
 gflags-devel \
 protobuf-devel \
 protobuf-compiler \
 leveldb-devel \
 snappy-devel \
 glog-devel

# 创建 cmake 符号链接
sudo ln -sf /usr/bin/cmake3 /usr/bin/cmake

2.3 编译 brpc

media-server 依赖 brpc，首先需要编译安装 brpc：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


# 克隆 brpc 源码
git clone https://github.com/apache/brpc.git
cd brpc

# 创建构建目录
mkdir build && cd build

# 配置编译选项
cmake .. \
 -DCMAKE_BUILD_TYPE=Release \
 -DWITH_GLOG=ON \
 -DWITH_DEBUG_SYMBOLS=OFF

# 编译（使用所有 CPU 核心）
make -j$(nproc)

# 安装到系统
sudo make install

# 更新动态链接库缓存
sudo ldconfig

2.4 编译 media-server

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# 克隆 media-server 源码
git clone https://github.com/brpc/media-server.git
cd media-server

# 创建构建目录
mkdir build && cd build

# 配置编译
cmake .. -DCMAKE_BUILD_TYPE=Release

# 编译
make -j$(nproc)

# 验证编译结果
ls -la media_server

三、快速开始

3.1 基础启动

启动源站服务器

1
2
3
4
5
6
7
8


# 基础启动（默认端口 8079）
./media_server

# 指定端口启动
./media_server --port=8080

# 启用详细日志
./media_server --v=1

启动边缘服务器

1
2


# 作为边缘服务器启动
./media_server --edge_server=true --origin_server="192.168.1.100:8079"

3.2 基础推流和拉流测试

使用 FFmpeg 推流

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# 推送测试视频流
ffmpeg -re -i test_video.mp4 \
 -c:v libx264 -preset veryfast -tune zerolatency \
 -c:a aac -ar 44100 \
 -f flv rtmp://localhost:8079/live/test

# 推送摄像头直播流
ffmpeg -f v4l2 -i /dev/video0 \
 -c:v libx264 -preset ultrafast \
 -c:a aac -f flv \
 rtmp://localhost:8079/live/webcam

# 推送音频流（用于播客）
ffmpeg -f alsa -i default \
 -c:a aac -b:a 128k \
 -f flv rtmp://localhost:8079/radio/music

播放流媒体

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# 播放 RTMP 流
ffplay rtmp://localhost:8079/live/test

# 播放 HTTP-FLV 流
ffplay http://localhost:8079/live/test.flv

# 播放 HLS 流
ffplay http://localhost:8079/live/test.m3u8

# 使用 VLC 播放器
vlc rtmp://localhost:8079/live/test

3.3 Web 播放器集成

HTML5 播放器示例

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69


<!DOCTYPE html>
<html>
<head>
 <title>brpc media-server 流媒体播放</title>
 <script src="https://cdn.jsdelivr.net/npm/hls.js@latest"></script>
</head>
<body>
 <h1>直播流播放器</h1>

 <!-- HLS 播放器 -->
 <div>
 <h3>HLS 播放器</h3>
 <video id="hlsPlayer" controls width="640" height="360">
 <source src="http://localhost:8079/live/test.m3u8" type="application/vnd.apple.mpegurl">
 </video>
 </div>

 <!-- HTTP-FLV 播放器 -->
 <div>
 <h3>FLV 播放器</h3>
 <video id="flvPlayer" controls width="640" height="360">
 <source src="http://localhost:8079/live/test.flv" type="video/x-flv">
 </video>
 </div>

 <script>
 // HLS.js 配置
 if (Hls.isSupported()) {
 var video = document.getElementById('hlsPlayer');
 var hls = new Hls({
 debug: true,
 enableWorker: false,
 lowLatencyMode: true,
 backBufferLength: 90
 });

 hls.loadSource('http://localhost:8079/live/test.m3u8');
 hls.attachMedia(video);

 hls.on(Hls.Events.MANIFEST_PARSED, function() {
 console.log('HLS manifest parsed, starting playback');
 video.play();
 });

 hls.on(Hls.Events.ERROR, function(event, data) {
 console.error('HLS error:', data);
 });
 }

 // 自动重连机制
 function setupAutoReconnect(videoElement, streamUrl) {
 videoElement.addEventListener('error', function() {
 console.log('播放错误，5秒后重试...');
 setTimeout(function() {
 videoElement.src = streamUrl + '?t=' + new Date().getTime();
 videoElement.load();
 videoElement.play();
 }, 5000);
 });
 }

 // 为播放器设置自动重连
 setupAutoReconnect(
 document.getElementById('flvPlayer'),
 'http://localhost:8079/live/test.flv'
 );
 </script>
</body>
</html>

四、高级配置

4.1 服务器配置

命令行参数详解

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


# 基础配置
./media_server \
 --port=8079 \  # 服务端口
 --idle_timeout_s=-1 \  # 连接空闲超时（-1表示不超时）
 --max_concurrency=0 \  # 最大并发连接数（0表示不限制）
 --internal_port=-1 \  # 内部状态端口（-1表示禁用）

# 边缘服务器配置
 --edge_server=true \  # 启用边缘模式
 --origin_server="origin:8079" \  # 源站地址

# 性能优化
 --worker_thread_num=8 \  # 工作线程数
 --io_thread_num=4 \  # IO线程数

# 日志配置
 --log_dir="./logs" \  # 日志目录
 --v=1 \  # 日志级别
 --logbufsecs=0 # 日志缓冲时间

配置文件示例

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


# 创建配置文件 media_server.conf
cat > media_server.conf << 'EOF'
# 基础配置
port = 8079
max_concurrency = 1000
worker_thread_num = 8

# 流媒体配置
rtmp_port = 1935
http_port = 8080
enable_hls = true
enable_flv = true

# HLS 配置
hls_segment_duration = 2
hls_window_size = 10
hls_low_latency = true

# 录制配置
enable_recording = false
recording_path = "/data/recordings"

# 安全配置
enable_auth = false
auth_secret = "your_secret_key"

# 日志配置
log_level = INFO
log_file = "/var/log/media_server.log"
EOF

# 使用配置文件启动
./media_server --flagfile=media_server.conf

4.2 流媒体配置优化

低延迟配置

1
2
3
4
5
6
7
8


# 低延迟直播配置
./media_server \
 --port=8079 \
 --rtmp_gop_cache=false \  # 禁用GOP缓存
 --hls_segment_duration=1 \  # 1秒分片
 --hls_window_size=3 \  # 3个分片窗口
 --buffer_time_ms=500 \  # 500ms缓冲
 --max_frame_queue_size=10 # 最大帧队列大小

高质量配置

1
2
3
4
5
6
7
8


# 高质量直播配置
./media_server \
 --port=8079 \
 --enable_audio_only=false \  # 启用视频
 --enable_video_only=false \  # 启用音频
 --hls_segment_duration=6 \  # 6秒分片
 --hls_window_size=6 \  # 6个分片窗口
 --buffer_time_ms=3000 # 3秒缓冲

4.3 集群部署配置

源站集群

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# 源站1
./media_server \
 --port=8079 \
 --cluster_id="origin_cluster" \
 --node_id="origin_1" \
 --redis_host="redis.example.com" \
 --redis_port=6379

# 源站2
./media_server \
 --port=8079 \
 --cluster_id="origin_cluster" \
 --node_id="origin_2" \
 --redis_host="redis.example.com" \
 --redis_port=6379

边缘节点部署

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# 边缘节点1（北京）
./media_server \
 --port=8079 \
 --edge_server=true \
 --origin_servers="origin1:8079,origin2:8079" \
 --region="beijing" \
 --cache_size_mb=1024

# 边缘节点2（上海）
./media_server \
 --port=8079 \
 --edge_server=true \
 --origin_servers="origin1:8079,origin2:8079" \
 --region="shanghai" \
 --cache_size_mb=1024

五、监控和管理

5.1 HTTP 状态接口

基础状态查询

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# 查看服务器状态
curl "http://localhost:8079/status"

# 查看活跃流列表
curl "http://localhost:8079/api/streams"

# 查看特定流信息
curl "http://localhost:8079/api/stream/live/test"

# 查看服务器统计
curl "http://localhost:8079/api/stats" | jq .

响应示例

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


{
 "server_info": {
 "version": "1.0.0",
 "start_time": "2025-09-18T15:30:00Z",
 "uptime_seconds": 3600
 },
 "streams": {
 "total": 10,
 "active": 8,
 "inactive": 2
 },
 "connections": {
 "total": 150,
 "rtmp": 50,
 "http_flv": 80,
 "hls": 20
 },
 "bandwidth": {
 "incoming_mbps": 45.2,
 "outgoing_mbps": 180.8
 }
}

5.2 Prometheus 监控集成

监控指标导出

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


// 自定义监控指标收集器
class MediaServerMetrics {
public:
 static std::string export_prometheus_metrics() {
 std::ostringstream metrics;

 // 基础指标
 metrics << "# HELP media_server_connections_total Total connections\n";
 metrics << "media_server_connections_total{type=\"rtmp\"} " << get_rtmp_connections() << "\n";
 metrics << "media_server_connections_total{type=\"http_flv\"} " << get_flv_connections() << "\n";
 metrics << "media_server_connections_total{type=\"hls\"} " << get_hls_connections() << "\n";

 // 流统计
 metrics << "# HELP media_server_streams_active Active streams count\n";
 metrics << "media_server_streams_active " << get_active_streams() << "\n";

 // 带宽统计
 metrics << "# HELP media_server_bandwidth_bytes Bandwidth usage in bytes/sec\n";
 metrics << "media_server_bandwidth_bytes{direction=\"incoming\"} " << get_incoming_bandwidth() << "\n";
 metrics << "media_server_bandwidth_bytes{direction=\"outgoing\"} " << get_outgoing_bandwidth() << "\n";

 // 性能指标
 metrics << "# HELP media_server_cpu_usage_percent CPU usage percentage\n";
 metrics << "media_server_cpu_usage_percent " << get_cpu_usage() << "\n";

 metrics << "# HELP media_server_memory_usage_bytes Memory usage in bytes\n";
 metrics << "media_server_memory_usage_bytes " << get_memory_usage() << "\n";

 return metrics.str();
 }

private:
 static int get_rtmp_connections() { return 50; }
 static int get_flv_connections() { return 80; }
 static int get_hls_connections() { return 20; }
 static int get_active_streams() { return 8; }
 static long get_incoming_bandwidth() { return 45200000; } // 字节/秒
 static long get_outgoing_bandwidth() { return 180800000; }
 static double get_cpu_usage() { return 45.2; }
 static long get_memory_usage() { return 512 * 1024 * 1024; }
};

HTTP 监控端点

1
2


# 添加监控端点到 media-server
curl "http://localhost:8079/metrics"

5.3 日志管理

日志配置

1
2
3
4
5
6
7


# 启动时指定日志配置
./media_server \
 --log_dir="/var/log/media_server" \
 --v=2 \  # 详细日志级别
 --max_log_size=100 \  # 最大日志文件大小(MB)
 --logbufsecs=0 \  # 立即刷新日志
 --logtostderr=false # 不输出到stderr

日志轮转

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# 配置 logrotate
sudo tee /etc/logrotate.d/media_server << 'EOF'
/var/log/media_server/*.log {
 daily
 missingok
 rotate 30
 compress
 delaycompress
 notifempty
 create 644 media_server media_server
 postrotate
 killall -USR1 media_server || true
 endscript
}
EOF

六、性能优化

6.1 系统级优化

内核参数调优

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


# 创建优化脚本
cat > optimize_system.sh << 'EOF'
#!/bin/bash

# 网络优化
echo 'net.core.rmem_max = 268435456' >> /etc/sysctl.conf
echo 'net.core.wmem_max = 268435456' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_rmem = 4096 16384 268435456' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_wmem = 4096 16384 268435456' >> /etc/sysctl.conf
echo 'net.core.netdev_max_backlog = 5000' >> /etc/sysctl.conf

# 文件描述符限制
echo 'fs.file-max = 1000000' >> /etc/sysctl.conf
echo '* soft nofile 1000000' >> /etc/security/limits.conf
echo '* hard nofile 1000000' >> /etc/security/limits.conf

# 应用配置
sysctl -p

echo "系统优化完成"
EOF

chmod +x optimize_system.sh
sudo ./optimize_system.sh

6.2 应用级优化

编译优化

1
2
3
4
5
6
7
8


# 使用优化编译选项
cmake .. \
 -DCMAKE_BUILD_TYPE=Release \
 -DCMAKE_CXX_FLAGS="-O3 -march=native -mtune=native" \
 -DWITH_DEBUG_SYMBOLS=OFF \
 -DWITH_PROFILING=OFF

make -j$(nproc)

运行时优化

1
2
3
4
5
6
7
8
9


# CPU 亲和性设置
taskset -c 0-7 ./media_server --worker_thread_num=8

# 内存锁定
ulimit -l unlimited
./media_server --lock_memory=true

# 实时调度优先级
sudo nice -n -10 ./media_server

6.3 负载均衡配置

Nginx 负载均衡

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49


# /etc/nginx/conf.d/media_server.conf
upstream media_servers {
 least_conn;
 server 192.168.1.10:8079 weight=3 max_fails=3 fail_timeout=30s;
 server 192.168.1.11:8079 weight=3 max_fails=3 fail_timeout=30s;
 server 192.168.1.12:8079 weight=2 max_fails=3 fail_timeout=30s;
}

# RTMP 代理配置
server {
 listen 1935;
 proxy_pass media_servers;
 proxy_timeout 1s;
 proxy_responses 1;
 proxy_bind $remote_addr transparent;
}

# HTTP-FLV 代理配置
server {
 listen 80;
 server_name stream.example.com;

 location /live/ {
 proxy_pass http://media_servers;
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

 # 流媒体优化
 proxy_buffering off;
 proxy_cache off;
 proxy_set_header Connection '';
 proxy_http_version 1.1;
 chunked_transfer_encoding off;
 }

 # HLS 文件缓存
 location ~* \.(m3u8|ts)$ {
 proxy_pass http://media_servers;
 proxy_cache media_cache;
 proxy_cache_valid 200 1m;
 proxy_cache_lock on;
 expires 1m;
 add_header Cache-Control "public, no-transform";
 }
}

# 缓存配置
proxy_cache_path /var/cache/nginx/media levels=1:2 keys_zone=media_cache:10m inactive=60m max_size=1g;

七、安全配置

7.1 访问控制

IP 白名单

1
2
3
4


# 启动时指定允许的IP范围
./media_server \
 --allowed_ips="192.168.1.0/24,10.0.0.0/8" \
 --block_private_ips=false

推流认证

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34


// 自定义认证处理器
class StreamAuthHandler {
public:
 // RTMP 推流认证
 static bool authenticate_publish(const std::string& app,
 const std::string& stream_name,
 const std::string& auth_token) {
 // 验证推流权限
 std::string expected_token = generate_token(app, stream_name);
 return auth_token == expected_token;
 }

 // 播放认证
 static bool authenticate_play(const std::string& app,
 const std::string& stream_name,
 const std::string& client_ip) {
 // 检查播放权限
 return is_allowed_to_play(client_ip, app, stream_name);
 }

private:
 static std::string generate_token(const std::string& app,
 const std::string& stream) {
 // 实现token生成逻辑
 return hash_with_secret(app + "/" + stream + "/" + std::to_string(time(nullptr)));
 }

 static bool is_allowed_to_play(const std::string& ip,
 const std::string& app,
 const std::string& stream) {
 // 实现播放权限检查
 return check_whitelist(ip) && check_stream_permissions(app, stream);
 }
};

7.2 HTTPS/SSL 配置

SSL 证书配置

1
2
3
4
5
6
7
8
9


# 生成自签名证书（测试用）
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes

# 启用 HTTPS
./media_server \
 --enable_ssl=true \
 --ssl_cert="cert.pem" \
 --ssl_key="key.pem" \
 --ssl_port=8443

Let’s Encrypt 证书

1
2
3
4
5
6
7
8
9


# 安装 certbot
sudo apt-get install certbot

# 获取证书
sudo certbot certonly --standalone -d stream.example.com

# 配置自动续期
sudo crontab -e
# 添加：0 3 * * * /usr/bin/certbot renew --quiet --post-hook "systemctl restart media_server"

八、实战案例

8.1 构建直播平台

架构设计

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


 ┌─────────────┐
 │ CDN │
 │ (边缘节点) │
 └─────────────┘
 │
 ┌─────────────┐
 │ Load │
 │ Balancer │
 └─────────────┘
 │
 ┌────────────────┼────────────────┐
 │ │ │
 ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
 │ Media │ │ Media │ │ Media │
 │ Server 1 │ │ Server 2 │ │ Server 3 │
 └─────────────┘ └─────────────┘ └─────────────┘
 │ │ │
 └────────────────┼────────────────┘
 │
 ┌─────────────┐
 │ Redis │
 │ (状态存储) │
 └─────────────┘

部署脚本

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93


#!/bin/bash

# 部署脚本 deploy_streaming_platform.sh

# 配置变量
MEDIA_SERVERS=("192.168.1.10" "192.168.1.11" "192.168.1.12")
REDIS_HOST="192.168.1.20"
DOMAIN="stream.example.com"

echo "部署流媒体平台..."

# 1. 部署 Redis
echo "配置 Redis..."
ssh root@${REDIS_HOST} << 'EOF'
# 安装 Redis
apt-get update && apt-get install -y redis-server

# 配置 Redis
cat > /etc/redis/redis.conf << 'REDIS_CONF'
bind 0.0.0.0
port 6379
maxmemory 2gb
maxmemory-policy allkeys-lru
save 900 1
REDIS_CONF

systemctl restart redis-server
systemctl enable redis-server
EOF

# 2. 部署 Media Servers
for i in "${!MEDIA_SERVERS[@]}"; do
 SERVER=${MEDIA_SERVERS[$i]}
 echo "部署 Media Server $((i+1)) 到 $SERVER..."

 ssh root@${SERVER} << EOF
# 创建部署目录
mkdir -p /opt/media_server
cd /opt/media_server

# 下载并编译 media-server
git clone https://github.com/brpc/media-server.git
cd media-server
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j\$(nproc)

# 创建配置文件
cat > /opt/media_server/media_server.conf << 'CONF'
port = 8079
max_concurrency = 1000
worker_thread_num = 8
redis_host = ${REDIS_HOST}
redis_port = 6379
cluster_id = streaming_cluster
node_id = node_$((i+1))
CONF

# 创建 systemd 服务
cat > /etc/systemd/system/media_server.service << 'SERVICE'
[Unit]
Description=Media Server
After=network.target

[Service]
Type=simple
User=media_server
WorkingDirectory=/opt/media_server/media-server/build
ExecStart=/opt/media_server/media-server/build/media_server --flagfile=/opt/media_server/media_server.conf
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
SERVICE

# 创建用户
useradd -r -s /bin/false media_server
chown -R media_server:media_server /opt/media_server

# 启动服务
systemctl daemon-reload
systemctl enable media_server
systemctl start media_server
EOF

done

# 3. 配置负载均衡器
echo "配置负载均衡器..."
# 这里可以配置 Nginx 或其他负载均衡器

echo "部署完成！"

8.2 实时互动直播

WebRTC 集成

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68


// WebRTC 推流客户端
class WebRTCStreamer {
 constructor(serverUrl) {
 this.serverUrl = serverUrl;
 this.localStream = null;
 this.pc = null;
 }

 async startStreaming() {
 try {
 // 获取摄像头和麦克风
 this.localStream = await navigator.mediaDevices.getUserMedia({
 video: {
 width: { ideal: 1280 },
 height: { ideal: 720 },
 frameRate: { ideal: 30 }
 },
 audio: {
 echoCancellation: true,
 noiseSuppression: true
 }
 });

 // 创建 RTCPeerConnection
 this.pc = new RTCPeerConnection({
 iceServers: [
 { urls: 'stun:stun.l.google.com:19302' }
 ]
 });

 // 添加本地流
 this.localStream.getTracks().forEach(track => {
 this.pc.addTrack(track, this.localStream);
 });

 // 创建 offer
 const offer = await this.pc.createOffer();
 await this.pc.setLocalDescription(offer);

 // 发送到服务器
 const response = await fetch(`${this.serverUrl}/webrtc/offer`, {
 method: 'POST',
 headers: { 'Content-Type': 'application/json' },
 body: JSON.stringify({ offer: offer })
 });

 const answer = await response.json();
 await this.pc.setRemoteDescription(answer);

 console.log('WebRTC streaming started');
 } catch (error) {
 console.error('Error starting WebRTC stream:', error);
 }
 }

 stopStreaming() {
 if (this.localStream) {
 this.localStream.getTracks().forEach(track => track.stop());
 }
 if (this.pc) {
 this.pc.close();
 }
 }
}

// 使用示例
const streamer = new WebRTCStreamer('http://localhost:8079');
streamer.startStreaming();

8.3 多码率自适应流

转码配置

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


# 创建多码率转码脚本
cat > transcode_multirate.sh << 'EOF'
#!/bin/bash

INPUT_STREAM="$1"
OUTPUT_BASE="$2"

# 转码为多种码率
ffmpeg -i "$INPUT_STREAM" \
 -c:v libx264 -preset veryfast \
 -map 0:v -map 0:a -map 0:v -map 0:a -map 0:v -map 0:a \
 \
 -s:v:0 1920x1080 -b:v:0 3000k -maxrate:v:0 3200k -bufsize:v:0 6000k \
 -s:v:1 1280x720 -b:v:1 1500k -maxrate:v:1 1650k -bufsize:v:1 3000k \
 -s:v:2 854x480 -b:v:2 800k -maxrate:v:2 880k -bufsize:v:2 1600k \
 \
 -c:a:0 aac -b:a:0 128k \
 -c:a:1 aac -b:a:1 128k \
 -c:a:2 aac -b:a:2 96k \
 \
 -f hls -hls_time 6 -hls_list_size 10 \
 -hls_segment_filename "${OUTPUT_BASE}_%v_%03d.ts" \
 -master_pl_name "master.m3u8" \
 -var_stream_map "v:0,a:0 v:1,a:1 v:2,a:2" \
 "${OUTPUT_BASE}_%v.m3u8"
EOF

chmod +x transcode_multirate.sh

# 使用
./transcode_multirate.sh "rtmp://localhost:8079/live/input" "/var/www/hls/stream"

九、故障排查

9.1 常见问题诊断

连接问题

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# 检查端口监听
netstat -tulpn | grep 8079

# 检查防火墙
sudo ufw status
sudo iptables -L

# 测试 RTMP 连接
ffmpeg -f lavfi -i testsrc=duration=10:size=320x240:rate=30 \
 -f flv rtmp://localhost:8079/live/test

# 检查服务器日志
tail -f /var/log/media_server/media_server.log

性能问题

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# 系统资源监控
top -p $(pgrep media_server)
iostat -x 1
sar -n DEV 1

# 网络监控
iftop -i eth0
ss -tuln

# 内存使用分析
pmap -x $(pgrep media_server)
valgrind --tool=massif ./media_server

9.2 调试工具

GDB 调试

1
2
3
4
5
6
7
8
9


# 编译 debug 版本
cmake .. -DCMAKE_BUILD_TYPE=Debug
make -j$(nproc)

# GDB 调试
gdb ./media_server
(gdb) set args --port=8079 --v=2
(gdb) run
(gdb) bt # 查看调用栈

性能分析

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# CPU 性能分析
perf record -g ./media_server --port=8079
perf report

# 内存泄漏检测
valgrind --leak-check=full --show-leak-kinds=all ./media_server

# 网络抓包分析
tcpdump -i any -w media_server.pcap port 8079
wireshark media_server.pcap

十、总结

brpc media-server 是一个功能强大、性能优异的流媒体服务器解决方案。通过本指南，你已经掌握了：

核心能力

多协议支持：RTMP、HTTP-FLV、HLS 等主流协议
高性能架构：基于 brpc 的高并发处理
灵活部署：支持源站和边缘节点部署
丰富配置：满足不同场景的配置需求

应用场景

直播平台：构建大规模直播服务
在线教育：实时互动教学
视频会议：企业级视频通信
IoT 流媒体：物联网设备视频传输

最佳实践

性能优化：系统调优和应用配置
安全防护：访问控制和加密传输
监控运维：完善的监控和日志系统
故障处理：快速诊断和恢复机制

brpc media-server 为构建企业级流媒体服务提供了坚实的基础，结合合适的架构设计和运维实践，可以构建出稳定、高效的流媒体平台。

]]>

braft 完全指南：从零掌握分布式一致性编程

lesshash@blog.lesshash.com/ (lesshash) — Thu, 18 Sep 2025 15:00:00 +0800

一、为什么需要 braft？

1.1 分布式系统的一致性挑战

在分布式系统中，数据一致性是最核心的挑战之一。当多个节点需要对某个状态达成一致时，面临诸多困难：

网络分区：节点间通信可能中断
节点故障：任何节点都可能随时宕机
消息乱序：网络延迟导致消息到达顺序不确定
脑裂问题：网络分区可能导致集群分裂

1.2 Raft 算法简介

Raft 是一个易于理解的分布式一致性算法，相比 Paxos 更加简单明了：

Leader 选举：在任意时刻，集群中最多只有一个 Leader
日志复制：Leader 负责接收客户端请求并复制到其他节点
安全性保证：确保已提交的日志条目不会丢失

1.3 braft 的优势

braft 是百度开源的 Raft 算法 C++ 实现，具有以下优势：

性能优势

高吞吐量：单集群可达 100万+ QPS
低延迟：Leader 选举通常在 200ms 内完成
批量处理：支持日志批量提交，提高效率

工程优势

生产就绪：在百度内部大规模使用，经过充分验证
易于集成：与 brpc 无缝集成，API 设计友好
可观测性：丰富的监控指标和调试工具

功能完整性

快照支持：支持数据快照，减少日志占用空间
成员变更：支持动态添加/删除节点
线性一致性读：支持强一致性读操作

二、快速入门

2.1 环境搭建

依赖安装

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Ubuntu/Debian
sudo apt-get update
sudo apt-get install -y build-essential cmake git
sudo apt-get install -y libssl-dev libprotobuf-dev protobuf-compiler
sudo apt-get install -y libgflags-dev libgoogle-glog-dev

# 安装 brpc（braft 依赖）
git clone https://github.com/apache/brpc.git
cd brpc && mkdir build && cd build
cmake .. && make -j$(nproc) && sudo make install

编译 braft

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# 克隆源码
git clone https://github.com/baidu/braft.git
cd braft

# 编译
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)

# 安装
sudo make install

2.2 第一个 braft 程序

我们来实现一个简单的分布式计数器，它能在多个节点间保持一致。

Step 1: 定义状态机

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160


#include <braft/raft.h>
#include <braft/util.h>
#include <braft/storage.h>

// 操作类型
enum OpType {
 OP_INCREMENT = 1,
 OP_DECREMENT = 2,
 OP_SET = 3
};

// 操作日志
struct CounterOp {
 OpType type;
 int64_t value;

 // 序列化
 std::string serialize() const {
 return std::to_string(type) + ":" + std::to_string(value);
 }

 // 反序列化
 static CounterOp deserialize(const std::string& data) {
 size_t pos = data.find(':');
 CounterOp op;
 op.type = static_cast<OpType>(std::stoi(data.substr(0, pos)));
 op.value = std::stoll(data.substr(pos + 1));
 return op;
 }
};

// 状态机实现
class CounterStateMachine : public braft::StateMachine {
public:
 CounterStateMachine() : _counter(0) {}

 // 应用日志到状态机
 void on_apply(braft::Iterator& iter) override {
 for (; iter.valid(); iter.next()) {
 braft::AsyncClosureGuard closure_guard(iter.done());

 if (iter.done()) {
 // 这是来自客户端的请求
 CounterOp op = CounterOp::deserialize(iter.data().to_string());
 int64_t old_value = _counter;

 switch (op.type) {
 case OP_INCREMENT:
 _counter += op.value;
 break;
 case OP_DECREMENT:
 _counter -= op.value;
 break;
 case OP_SET:
 _counter = op.value;
 break;
 }

 LOG(INFO) << "Applied operation: " << op.serialize()
 << ", counter: " << old_value << " -> " << _counter;

 // 返回结果给客户端
 if (iter.done()) {
 static_cast<CounterClosure*>(iter.done())->set_result(_counter);
 }
 } else {
 // 这是从快照恢复或日志回放
 CounterOp op = CounterOp::deserialize(iter.data().to_string());
 switch (op.type) {
 case OP_INCREMENT: _counter += op.value; break;
 case OP_DECREMENT: _counter -= op.value; break;
 case OP_SET: _counter = op.value; break;
 }
 }
 }
 }

 // 保存快照
 void on_snapshot_save(braft::SnapshotWriter* writer, braft::Closure* done) override {
 brpc::ClosureGuard done_guard(done);

 // 保存计数器值到快照
 std::string snapshot_path = writer->get_path() + "/counter";
 std::ofstream file(snapshot_path);
 if (!file.is_open()) {
 LOG(ERROR) << "Failed to open snapshot file: " << snapshot_path;
 done->status().set_error(EIO, "Failed to save snapshot");
 return;
 }

 file << _counter;
 file.close();

 // 添加文件到快照
 if (writer->add_file("counter") != 0) {
 LOG(ERROR) << "Failed to add file to snapshot";
 done->status().set_error(EIO, "Failed to add file to snapshot");
 return;
 }

 LOG(INFO) << "Saved snapshot with counter: " << _counter;
 }

 // 加载快照
 int on_snapshot_load(braft::SnapshotReader* reader) override {
 if (!reader->list_files().count("counter")) {
 LOG(ERROR) << "Counter file not found in snapshot";
 return -1;
 }

 std::string snapshot_path = reader->get_path() + "/counter";
 std::ifstream file(snapshot_path);
 if (!file.is_open()) {
 LOG(ERROR) << "Failed to open snapshot file: " << snapshot_path;
 return -1;
 }

 file >> _counter;
 file.close();

 LOG(INFO) << "Loaded snapshot with counter: " << _counter;
 return 0;
 }

 // 处理 Leader 变更
 void on_leader_start(int64_t term) override {
 LOG(INFO) << "Became leader at term: " << term;
 }

 void on_leader_stop(const butil::Status& status) override {
 LOG(INFO) << "Stepped down as leader: " << status;
 }

 // 获取当前计数器值
 int64_t get_counter() const { return _counter; }

private:
 int64_t _counter;
};

// 异步回调
class CounterClosure : public braft::Closure {
public:
 CounterClosure() : _result(0) {}

 void Run() override {
 if (status().ok()) {
 LOG(INFO) << "Operation succeeded, result: " << _result;
 } else {
 LOG(ERROR) << "Operation failed: " << status();
 }
 delete this;
 }

 void set_result(int64_t result) { _result = result; }
 int64_t get_result() const { return _result; }

private:
 int64_t _result;
};

Step 2: 实现 Raft 节点

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138


class CounterServer {
public:
 CounterServer() : _node(nullptr), _state_machine(new CounterStateMachine()) {}

 ~CounterServer() {
 if (_node) {
 _node->shutdown(nullptr);
 _node->join();
 delete _node;
 }
 delete _state_machine;
 }

 // 启动节点
 int start(const std::string& listen_addr,
 const std::string& raft_addr,
 const std::string& peers,
 const std::string& data_dir) {

 // 创建 RPC 服务器
 if (_server.Start(listen_addr.c_str(), nullptr) != 0) {
 LOG(ERROR) << "Failed to start RPC server";
 return -1;
 }

 // 配置 Raft 节点
 braft::NodeOptions node_options;

 // 设置状态机
 node_options.fsm = _state_machine;

 // 设置存储路径
 node_options.log_uri = data_dir + "/log";
 node_options.raft_meta_uri = data_dir + "/raft_meta";
 node_options.snapshot_uri = data_dir + "/snapshot";

 // 设置选举超时（毫秒）
 node_options.election_timeout_ms = 5000;

 // 设置快照间隔
 node_options.snapshot_interval_s = 30;

 // 禁用 CLI（命令行接口），生产环境中通常启用
 node_options.disable_cli = false;

 // 创建 Raft 节点
 _node = new braft::Node("counter_group", braft::PeerId(raft_addr));

 if (_node->init(node_options) != 0) {
 LOG(ERROR) << "Failed to init raft node";
 return -1;
 }

 // 如果指定了 peers，则作为集群成员启动
 if (!peers.empty()) {
 std::vector<braft::PeerId> peer_list;
 butil::StringSplitter sp(peers.c_str(), ',');
 for (; sp; ++sp) {
 braft::PeerId peer(sp.field(), sp.length());
 peer_list.push_back(peer);
 }

 braft::Configuration conf(peer_list);
 if (_node->bootstrap(conf) != 0) {
 LOG(ERROR) << "Failed to bootstrap raft node";
 return -1;
 }
 }

 LOG(INFO) << "Counter server started at " << listen_addr
 << ", raft at " << raft_addr;
 return 0;
 }

 // 执行操作
 void execute_operation(const CounterOp& op, CounterClosure* closure) {
 if (!_node->is_leader()) {
 closure->status().set_error(EPERM, "Not leader");
 closure->Run();
 return;
 }

 // 构造日志条目
 butil::IOBuf log_data;
 log_data.append(op.serialize());

 // 提交到 Raft
 braft::Task task;
 task.data = &log_data;
 task.done = closure;

 _node->apply(task);
 }

 // 获取当前值（强一致性读）
 int64_t get_counter_consistent() {
 if (!_node->is_leader()) {
 return -1; // 只有 Leader 才能提供强一致性读
 }

 // 通过 read_index 实现线性一致性读
 braft::ReadIndexClosure* read_closure =
 new braft::ReadIndexClosure([this](const braft::ReadIndexStatus& status) {
 if (status.status.ok()) {
 // 读取状态机数据
 return _state_machine->get_counter();
 }
 return (int64_t)-1;
 });

 _node->read_index(butil::IOBuf(), read_closure);
 // 注意：实际应用中需要等待回调完成
 return _state_machine->get_counter();
 }

 // 获取当前值（非强一致性读）
 int64_t get_counter_local() {
 return _state_machine->get_counter();
 }

 // 检查是否为 Leader
 bool is_leader() const {
 return _node && _node->is_leader();
 }

 // 获取 Leader 信息
 braft::PeerId get_leader() const {
 if (_node) {
 return _node->leader_id();
 }
 return braft::PeerId();
 }

private:
 brpc::Server _server;
 braft::Node* _node;
 CounterStateMachine* _state_machine;
};

Step 3: 实现客户端

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


// 基于 brpc 的客户端
class CounterClient {
public:
 CounterClient() {}

 // 连接到服务器
 int connect(const std::string& server_addr) {
 brpc::ChannelOptions options;
 options.protocol = brpc::PROTOCOL_BAIDU_STD;
 options.timeout_ms = 1000;
 options.max_retry = 3;

 if (_channel.Init(server_addr.c_str(), &options) != 0) {
 LOG(ERROR) << "Failed to connect to " << server_addr;
 return -1;
 }

 return 0;
 }

 // 增加计数器
 bool increment(int64_t value) {
 CounterOp op{OP_INCREMENT, value};
 return execute_operation(op);
 }

 // 减少计数器
 bool decrement(int64_t value) {
 CounterOp op{OP_DECREMENT, value};
 return execute_operation(op);
 }

 // 设置计数器值
 bool set(int64_t value) {
 CounterOp op{OP_SET, value};
 return execute_operation(op);
 }

 // 获取计数器值
 int64_t get() {
 // 这里简化为直接调用服务端方法
 // 实际应用中需要通过 RPC 调用
 return 0;
 }

private:
 bool execute_operation(const CounterOp& op) {
 // 实际实现需要定义 protobuf 接口并通过 RPC 调用
 // 这里只是示例框架
 return true;
 }

private:
 brpc::Channel _channel;
};

Step 4: 主程序

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75


#include <iostream>
#include <signal.h>
#include <gflags/gflags.h>

DEFINE_string(listen_addr, "127.0.0.1:8080", "Listen address for RPC");
DEFINE_string(raft_addr, "127.0.0.1:8081", "Raft listen address");
DEFINE_string(peers, "", "Initial peers, separated by comma");
DEFINE_string(data_dir, "./data", "Data directory");

static CounterServer* g_server = nullptr;

void signal_handler(int sig) {
 if (g_server) {
 LOG(INFO) << "Received signal " << sig << ", shutting down...";
 delete g_server;
 g_server = nullptr;
 exit(0);
 }
}

int main(int argc, char* argv[]) {
 google::ParseCommandLineFlags(&argc, &argv, true);

 // 设置信号处理
 signal(SIGINT, signal_handler);
 signal(SIGTERM, signal_handler);

 // 创建数据目录
 if (system(("mkdir -p " + FLAGS_data_dir).c_str()) != 0) {
 LOG(ERROR) << "Failed to create data directory";
 return -1;
 }

 // 启动服务器
 g_server = new CounterServer();
 if (g_server->start(FLAGS_listen_addr, FLAGS_raft_addr,
 FLAGS_peers, FLAGS_data_dir) != 0) {
 LOG(ERROR) << "Failed to start counter server";
 return -1;
 }

 // 等待用户输入
 std::string line;
 while (std::getline(std::cin, line)) {
 if (line == "quit" || line == "exit") {
 break;
 }

 if (line == "status") {
 std::cout << "Leader: " << g_server->is_leader() << std::endl;
 std::cout << "Counter: " << g_server->get_counter_local() << std::endl;
 std::cout << "Leader ID: " << g_server->get_leader() << std::endl;
 } else if (line.substr(0, 4) == "inc ") {
 int64_t value = std::stoll(line.substr(4));
 CounterClosure* closure = new CounterClosure();
 CounterOp op{OP_INCREMENT, value};
 g_server->execute_operation(op, closure);
 } else if (line.substr(0, 4) == "dec ") {
 int64_t value = std::stoll(line.substr(4));
 CounterClosure* closure = new CounterClosure();
 CounterOp op{OP_DECREMENT, value};
 g_server->execute_operation(op, closure);
 } else if (line.substr(0, 4) == "set ") {
 int64_t value = std::stoll(line.substr(4));
 CounterClosure* closure = new CounterClosure();
 CounterOp op{OP_SET, value};
 g_server->execute_operation(op, closure);
 } else {
 std::cout << "Commands: status, inc <value>, dec <value>, set <value>, quit" << std::endl;
 }
 }

 delete g_server;
 return 0;
}

2.3 编译和运行

创建 CMakeLists.txt

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


cmake_minimum_required(VERSION 3.10)
project(counter_example)

set(CMAKE_CXX_STANDARD 11)

find_package(PkgConfig REQUIRED)
find_package(Protobuf REQUIRED)
find_package(gflags REQUIRED)

# 查找 brpc 和 braft
find_library(BRPC_LIB brpc)
find_library(BRAFT_LIB braft)

add_executable(counter_server
 counter_server.cpp
)

target_link_libraries(counter_server
 ${BRAFT_LIB}
 ${BRPC_LIB}
 ${Protobuf_LIBRARIES}
 gflags
 ssl
 crypto
 dl
 z
)

编译运行

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# 编译
mkdir build && cd build
cmake .. && make

# 启动第一个节点（作为引导节点）
./counter_server --listen_addr=127.0.0.1:8080 --raft_addr=127.0.0.1:8081 \
 --peers=127.0.0.1:8081,127.0.0.1:8082,127.0.0.1:8083 \
 --data_dir=./node1

# 启动第二个节点
./counter_server --listen_addr=127.0.0.1:8090 --raft_addr=127.0.0.1:8082 \
 --data_dir=./node2

# 启动第三个节点
./counter_server --listen_addr=127.0.0.1:8100 --raft_addr=127.0.0.1:8083 \
 --data_dir=./node3

三、核心概念深入

3.1 Raft 状态机

3.1.1 状态机接口详解

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


class StateMachine {
public:
 // 应用日志条目到状态机
 virtual void on_apply(braft::Iterator& iter) = 0;

 // 保存快照
 virtual void on_snapshot_save(braft::SnapshotWriter* writer,
 braft::Closure* done) = 0;

 // 加载快照
 virtual int on_snapshot_load(braft::SnapshotReader* reader) = 0;

 // Leader 选举成功回调
 virtual void on_leader_start(int64_t term) {}

 // Leader 退位回调
 virtual void on_leader_stop(const butil::Status& status) {}

 // 启动回调
 virtual void on_start_following(const braft::LeaderChangeContext& ctx) {}

 // 停止 following 回调
 virtual void on_stop_following(const braft::LeaderChangeContext& ctx) {}

 // 配置变更回调
 virtual void on_configuration_committed(const braft::Configuration& conf,
 int64_t index) {}

 // 错误回调
 virtual void on_error(const braft::Error& e) {}
};

3.1.2 高级状态机示例

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272


// 支持多种操作的键值存储状态机
class KVStateMachine : public braft::StateMachine {
public:
 enum Operation {
 OP_PUT = 1,
 OP_DELETE = 2,
 OP_CAS = 3 // Compare-And-Swap
 };

 struct LogEntry {
 Operation op;
 std::string key;
 std::string value;
 std::string expected_value; // for CAS

 std::string serialize() const {
 butil::IOBuf buf;
 buf.append(reinterpret_cast<const char*>(&op), sizeof(op));

 uint32_t key_len = key.length();
 buf.append(reinterpret_cast<const char*>(&key_len), sizeof(key_len));
 buf.append(key);

 uint32_t value_len = value.length();
 buf.append(reinterpret_cast<const char*>(&value_len), sizeof(value_len));
 buf.append(value);

 if (op == OP_CAS) {
 uint32_t expected_len = expected_value.length();
 buf.append(reinterpret_cast<const char*>(&expected_len), sizeof(expected_len));
 buf.append(expected_value);
 }

 return buf.to_string();
 }

 static LogEntry deserialize(const std::string& data) {
 LogEntry entry;
 const char* ptr = data.c_str();
 size_t offset = 0;

 // 读取操作类型
 entry.op = *reinterpret_cast<const Operation*>(ptr + offset);
 offset += sizeof(Operation);

 // 读取 key
 uint32_t key_len = *reinterpret_cast<const uint32_t*>(ptr + offset);
 offset += sizeof(uint32_t);
 entry.key = std::string(ptr + offset, key_len);
 offset += key_len;

 // 读取 value
 uint32_t value_len = *reinterpret_cast<const uint32_t*>(ptr + offset);
 offset += sizeof(uint32_t);
 entry.value = std::string(ptr + offset, value_len);
 offset += value_len;

 // 如果是 CAS 操作，读取期望值
 if (entry.op == OP_CAS && offset < data.length()) {
 uint32_t expected_len = *reinterpret_cast<const uint32_t*>(ptr + offset);
 offset += sizeof(uint32_t);
 entry.expected_value = std::string(ptr + offset, expected_len);
 }

 return entry;
 }
 };

private:
 std::unordered_map<std::string, std::string> _kv_store;
 mutable bthread::Mutex _mutex;
 std::atomic<int64_t> _applied_index{0};

public:
 void on_apply(braft::Iterator& iter) override {
 for (; iter.valid(); iter.next()) {
 braft::AsyncClosureGuard closure_guard(iter.done());

 _applied_index.store(iter.index());

 LogEntry entry = LogEntry::deserialize(iter.data().to_string());
 bool success = false;
 std::string old_value;

 {
 std::unique_lock<bthread::Mutex> lock(_mutex);

 switch (entry.op) {
 case OP_PUT:
 old_value = _kv_store[entry.key];
 _kv_store[entry.key] = entry.value;
 success = true;
 break;

 case OP_DELETE:
 if (_kv_store.count(entry.key)) {
 old_value = _kv_store[entry.key];
 _kv_store.erase(entry.key);
 success = true;
 }
 break;

 case OP_CAS:
 if (_kv_store.count(entry.key) &&
 _kv_store[entry.key] == entry.expected_value) {
 old_value = _kv_store[entry.key];
 _kv_store[entry.key] = entry.value;
 success = true;
 }
 break;
 }
 }

 LOG(INFO) << "Applied " << entry.key << ": " << old_value
 << " -> " << entry.value << ", success: " << success;

 // 返回结果给客户端
 if (iter.done()) {
 KVClosure* kv_closure = dynamic_cast<KVClosure*>(iter.done());
 if (kv_closure) {
 kv_closure->set_result(success, old_value);
 }
 }
 }
 }

 void on_snapshot_save(braft::SnapshotWriter* writer, braft::Closure* done) override {
 brpc::ClosureGuard done_guard(done);

 std::string snapshot_path = writer->get_path() + "/kv_data";
 std::ofstream file(snapshot_path, std::ios::binary);
 if (!file.is_open()) {
 LOG(ERROR) << "Failed to open snapshot file";
 done->status().set_error(EIO, "Failed to save snapshot");
 return;
 }

 {
 std::unique_lock<bthread::Mutex> lock(_mutex);

 // 保存当前应用的索引
 file.write(reinterpret_cast<const char*>(&_applied_index), sizeof(_applied_index));

 // 保存键值对数量
 uint64_t count = _kv_store.size();
 file.write(reinterpret_cast<const char*>(&count), sizeof(count));

 // 保存所有键值对
 for (const auto& pair : _kv_store) {
 uint32_t key_len = pair.first.length();
 file.write(reinterpret_cast<const char*>(&key_len), sizeof(key_len));
 file.write(pair.first.c_str(), key_len);

 uint32_t value_len = pair.second.length();
 file.write(reinterpret_cast<const char*>(&value_len), sizeof(value_len));
 file.write(pair.second.c_str(), value_len);
 }
 }

 file.close();

 if (writer->add_file("kv_data") != 0) {
 LOG(ERROR) << "Failed to add file to snapshot";
 done->status().set_error(EIO, "Failed to add file");
 return;
 }

 LOG(INFO) << "Saved snapshot with " << _kv_store.size()
 << " keys at index " << _applied_index.load();
 }

 int on_snapshot_load(braft::SnapshotReader* reader) override {
 if (!reader->list_files().count("kv_data")) {
 LOG(ERROR) << "kv_data not found in snapshot";
 return -1;
 }

 std::string snapshot_path = reader->get_path() + "/kv_data";
 std::ifstream file(snapshot_path, std::ios::binary);
 if (!file.is_open()) {
 LOG(ERROR) << "Failed to open snapshot file";
 return -1;
 }

 {
 std::unique_lock<bthread::Mutex> lock(_mutex);
 _kv_store.clear();

 // 读取应用索引
 int64_t applied_index;
 file.read(reinterpret_cast<char*>(&applied_index), sizeof(applied_index));
 _applied_index.store(applied_index);

 // 读取键值对数量
 uint64_t count;
 file.read(reinterpret_cast<char*>(&count), sizeof(count));

 // 读取所有键值对
 for (uint64_t i = 0; i < count; ++i) {
 uint32_t key_len;
 file.read(reinterpret_cast<char*>(&key_len), sizeof(key_len));
 std::string key(key_len, '\0');
 file.read(&key[0], key_len);

 uint32_t value_len;
 file.read(reinterpret_cast<char*>(&value_len), sizeof(value_len));
 std::string value(value_len, '\0');
 file.read(&value[0], value_len);

 _kv_store[key] = value;
 }
 }

 file.close();

 LOG(INFO) << "Loaded snapshot with " << _kv_store.size()
 << " keys at index " << _applied_index.load();
 return 0;
 }

 // 获取值
 bool get(const std::string& key, std::string* value) const {
 std::unique_lock<bthread::Mutex> lock(_mutex);
 auto it = _kv_store.find(key);
 if (it != _kv_store.end()) {
 *value = it->second;
 return true;
 }
 return false;
 }

 // 获取所有键
 std::vector<std::string> list_keys() const {
 std::unique_lock<bthread::Mutex> lock(_mutex);
 std::vector<std::string> keys;
 for (const auto& pair : _kv_store) {
 keys.push_back(pair.first);
 }
 return keys;
 }

 size_t size() const {
 std::unique_lock<bthread::Mutex> lock(_mutex);
 return _kv_store.size();
 }
};

// KV 操作的回调
class KVClosure : public braft::Closure {
public:
 void Run() override {
 if (status().ok()) {
 LOG(INFO) << "KV operation succeeded, result: " << _success
 << ", old_value: " << _old_value;
 } else {
 LOG(ERROR) << "KV operation failed: " << status();
 }
 delete this;
 }

 void set_result(bool success, const std::string& old_value) {
 _success = success;
 _old_value = old_value;
 }

 bool get_success() const { return _success; }
 const std::string& get_old_value() const { return _old_value; }

private:
 bool _success = false;
 std::string _old_value;
};

3.2 配置管理

3.2.1 动态成员变更

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142


class ConfigurationManager {
public:
 // 添加节点
 static void add_peer(braft::Node* node, const std::string& peer_addr,
 braft::Closure* done) {
 braft::Configuration new_conf;
 node->list_peers(&new_conf);

 braft::PeerId peer_id(peer_addr);
 new_conf.add_peer(peer_id);

 LOG(INFO) << "Adding peer: " << peer_addr;
 node->change_peers(new_conf, done);
 }

 // 移除节点
 static void remove_peer(braft::Node* node, const std::string& peer_addr,
 braft::Closure* done) {
 braft::Configuration new_conf;
 node->list_peers(&new_conf);

 braft::PeerId peer_id(peer_addr);
 new_conf.remove_peer(peer_id);

 LOG(INFO) << "Removing peer: " << peer_addr;
 node->change_peers(new_conf, done);
 }

 // 查看当前配置
 static std::vector<std::string> list_peers(braft::Node* node) {
 braft::Configuration conf;
 node->list_peers(&conf);

 std::vector<std::string> peers;
 for (auto it = conf.begin(); it != conf.end(); ++it) {
 peers.push_back(it->to_string());
 }
 return peers;
 }

 // 安全的配置变更（逐个添加/删除）
 static void safe_change_peers(braft::Node* node,
 const std::vector<std::string>& target_peers,
 braft::Closure* final_done) {
 // 获取当前配置
 braft::Configuration current_conf;
 node->list_peers(&current_conf);

 std::set<std::string> current_peers;
 for (auto it = current_conf.begin(); it != current_conf.end(); ++it) {
 current_peers.insert(it->to_string());
 }

 std::set<std::string> target_set(target_peers.begin(), target_peers.end());

 // 找出需要添加和删除的节点
 std::vector<std::string> to_add, to_remove;

 for (const auto& peer : target_set) {
 if (current_peers.find(peer) == current_peers.end()) {
 to_add.push_back(peer);
 }
 }

 for (const auto& peer : current_peers) {
 if (target_set.find(peer) == target_set.end()) {
 to_remove.push_back(peer);
 }
 }

 // 创建批量变更管理器
 BatchConfigChange* batch = new BatchConfigChange(node, to_add, to_remove, final_done);
 batch->start();
 }

private:
 // 批量配置变更管理器
 class BatchConfigChange {
 public:
 BatchConfigChange(braft::Node* node,
 const std::vector<std::string>& to_add,
 const std::vector<std::string>& to_remove,
 braft::Closure* done)
 : _node(node), _to_add(to_add), _to_remove(to_remove),
 _final_done(done), _current_index(0) {}

 void start() {
 if (!_to_add.empty()) {
 process_add();
 } else if (!_to_remove.empty()) {
 process_remove();
 } else {
 finish();
 }
 }

 private:
 void process_add() {
 if (_current_index >= _to_add.size()) {
 _current_index = 0;
 process_remove();
 return;
 }

 braft::Closure* done = new braft::NewCallback(this, &BatchConfigChange::on_add_done);
 add_peer(_node, _to_add[_current_index], done);
 }

 void process_remove() {
 if (_current_index >= _to_remove.size()) {
 finish();
 return;
 }

 braft::Closure* done = new braft::NewCallback(this, &BatchConfigChange::on_remove_done);
 remove_peer(_node, _to_remove[_current_index], done);
 }

 void on_add_done() {
 ++_current_index;
 process_add();
 }

 void on_remove_done() {
 ++_current_index;
 process_remove();
 }

 void finish() {
 if (_final_done) {
 _final_done->Run();
 }
 delete this;
 }

 braft::Node* _node;
 std::vector<std::string> _to_add;
 std::vector<std::string> _to_remove;
 braft::Closure* _final_done;
 size_t _current_index;
 };
};

3.3 日志压缩和快照

3.3.1 智能快照策略

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122


class SmartSnapshotManager {
public:
 struct SnapshotOptions {
 int64_t max_log_size = 100 * 1024 * 1024; // 100MB
 int64_t max_log_entries = 100000; // 10万条
 int32_t snapshot_interval_s = 3600; // 1小时
 int32_t min_snapshot_interval_s = 300; // 最小5分钟
 double log_growth_threshold = 2.0; // 日志增长2倍触发快照
 };

 SmartSnapshotManager(braft::Node* node, const SnapshotOptions& options)
 : _node(node), _options(options), _last_snapshot_time(0),
 _last_snapshot_log_size(0) {}

 // 检查是否需要创建快照
 bool should_create_snapshot() {
 time_t now = time(nullptr);

 // 获取当前日志信息
 int64_t current_log_size = get_log_size();
 int64_t current_log_entries = get_log_entries();

 // 检查时间间隔
 if (now - _last_snapshot_time >= _options.snapshot_interval_s) {
 LOG(INFO) << "Snapshot triggered by time interval";
 return true;
 }

 // 检查日志大小
 if (current_log_size >= _options.max_log_size) {
 LOG(INFO) << "Snapshot triggered by log size: " << current_log_size;
 return true;
 }

 // 检查日志条目数
 if (current_log_entries >= _options.max_log_entries) {
 LOG(INFO) << "Snapshot triggered by log entries: " << current_log_entries;
 return true;
 }

 // 检查日志增长率
 if (_last_snapshot_log_size > 0 &&
 current_log_size >= _last_snapshot_log_size * _options.log_growth_threshold &&
 now - _last_snapshot_time >= _options.min_snapshot_interval_s) {
 LOG(INFO) << "Snapshot triggered by log growth: "
 << _last_snapshot_log_size << " -> " << current_log_size;
 return true;
 }

 return false;
 }

 // 创建快照
 void create_snapshot() {
 if (_creating_snapshot.load()) {
 LOG(WARNING) << "Snapshot creation already in progress";
 return;
 }

 _creating_snapshot.store(true);

 SnapshotClosure* done = new SnapshotClosure([this](bool success) {
 _creating_snapshot.store(false);
 if (success) {
 _last_snapshot_time = time(nullptr);
 _last_snapshot_log_size = get_log_size();
 LOG(INFO) << "Snapshot created successfully";
 } else {
 LOG(ERROR) << "Failed to create snapshot";
 }
 });

 _node->snapshot(done);
 }

 // 定期检查（应该在单独线程中调用）
 void periodic_check() {
 while (!_stopped.load()) {
 if (should_create_snapshot()) {
 create_snapshot();
 }
 bthread_usleep(60 * 1000 * 1000); // 每分钟检查一次
 }
 }

 void stop() {
 _stopped.store(true);
 }

private:
 int64_t get_log_size() {
 // 实际实现需要从 braft 获取日志大小
 // 这里返回估算值
 return 0;
 }

 int64_t get_log_entries() {
 // 实际实现需要从 braft 获取日志条目数
 return 0;
 }

 class SnapshotClosure : public braft::Closure {
 public:
 SnapshotClosure(std::function<void(bool)> callback)
 : _callback(callback) {}

 void Run() override {
 _callback(status().ok());
 delete this;
 }

 private:
 std::function<void(bool)> _callback;
 };

 braft::Node* _node;
 SnapshotOptions _options;
 std::atomic<bool> _creating_snapshot{false};
 std::atomic<bool> _stopped{false};
 time_t _last_snapshot_time;
 int64_t _last_snapshot_log_size;
};

3.4 性能优化

3.4.1 批量提交优化

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167


class BatchCommitter {
public:
 struct BatchOptions {
 int32_t max_batch_size = 1000; // 最大批量大小
 int32_t max_batch_bytes = 1024 * 1024; // 最大批量字节数
 int32_t batch_timeout_ms = 10; // 批量超时
 int32_t max_inflight_batches = 10; // 最大并发批次
 };

 BatchCommitter(braft::Node* node, const BatchOptions& options)
 : _node(node), _options(options), _inflight_batches(0) {

 // 启动批量处理线程
 _batch_thread = std::thread([this] { batch_worker(); });
 }

 ~BatchCommitter() {
 _stopped.store(true);
 _cv.notify_all();
 if (_batch_thread.joinable()) {
 _batch_thread.join();
 }
 }

 // 提交单个操作
 void submit(const butil::IOBuf& data, braft::Closure* done) {
 PendingOperation op{data, done, butil::gettimeofday_us()};

 {
 std::unique_lock<std::mutex> lock(_mutex);
 _pending_ops.push_back(std::move(op));
 }

 _cv.notify_one();
 }

private:
 struct PendingOperation {
 butil::IOBuf data;
 braft::Closure* done;
 int64_t submit_time_us;
 };

 struct Batch {
 std::vector<PendingOperation> operations;
 butil::IOBuf combined_data;
 int64_t total_bytes = 0;
 };

 void batch_worker() {
 while (!_stopped.load()) {
 std::unique_lock<std::mutex> lock(_mutex);

 // 等待操作或超时
 _cv.wait_for(lock, std::chrono::milliseconds(_options.batch_timeout_ms),
 [this] { return !_pending_ops.empty() || _stopped.load(); });

 if (_stopped.load()) break;

 if (_pending_ops.empty()) continue;

 // 检查是否可以创建新批次
 if (_inflight_batches.load() >= _options.max_inflight_batches) {
 continue;
 }

 // 构建批次
 Batch batch = build_batch();
 lock.unlock();

 if (!batch.operations.empty()) {
 submit_batch(std::move(batch));
 }
 }
 }

 Batch build_batch() {
 Batch batch;
 int64_t now_us = butil::gettimeofday_us();

 auto it = _pending_ops.begin();
 while (it != _pending_ops.end() &&
 batch.operations.size() < _options.max_batch_size &&
 batch.total_bytes < _options.max_batch_bytes) {

 // 检查是否超时（强制提交）
 bool timeout = (now_us - it->submit_time_us) >=
 (_options.batch_timeout_ms * 1000);

 if (!timeout && batch.operations.size() > 0 &&
 batch.total_bytes + it->data.size() > _options.max_batch_bytes) {
 break; // 避免超过大小限制
 }

 batch.operations.push_back(std::move(*it));
 batch.total_bytes += it->data.size();

 // 合并数据
 batch.combined_data.append(it->data);

 it = _pending_ops.erase(it);

 if (timeout) break; // 超时操作优先处理
 }

 return batch;
 }

 void submit_batch(Batch batch) {
 _inflight_batches.fetch_add(1);

 // 创建批量回调
 BatchClosure* done = new BatchClosure(
 std::move(batch.operations),
 [this](bool success) {
 _inflight_batches.fetch_sub(1);
 }
 );

 // 提交到 Raft
 braft::Task task;
 task.data = &batch.combined_data;
 task.done = done;

 _node->apply(task);

 LOG(INFO) << "Submitted batch with " << batch.operations.size()
 << " operations, " << batch.total_bytes << " bytes";
 }

 class BatchClosure : public braft::Closure {
 public:
 BatchClosure(std::vector<PendingOperation> operations,
 std::function<void(bool)> callback)
 : _operations(std::move(operations)), _callback(callback) {}

 void Run() override {
 bool success = status().ok();

 // 调用所有原始回调
 for (auto& op : _operations) {
 if (op.done) {
 op.done->status() = status();
 op.done->Run();
 }
 }

 _callback(success);
 delete this;
 }

 private:
 std::vector<PendingOperation> _operations;
 std::function<void(bool)> _callback;
 };

 braft::Node* _node;
 BatchOptions _options;

 std::mutex _mutex;
 std::condition_variable _cv;
 std::vector<PendingOperation> _pending_ops;

 std::atomic<bool> _stopped{false};
 std::atomic<int32_t> _inflight_batches{0};
 std::thread _batch_thread;
};

3.4.2 读优化

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131


class ReadOptimizer {
public:
 // 线性一致性读
 class LinearizableReader {
 public:
 LinearizableReader(braft::Node* node) : _node(node) {}

 // 执行线性一致性读
 void read(const std::function<void(bool, const std::string&)>& callback) {
 if (!_node->is_leader()) {
 callback(false, "Not leader");
 return;
 }

 ReadClosure* done = new ReadClosure(callback);
 _node->read_index(butil::IOBuf(), done);
 }

 private:
 class ReadClosure : public braft::ReadIndexClosure {
 public:
 ReadClosure(std::function<void(bool, const std::string&)> callback)
 : _callback(callback) {}

 void Run() override {
 if (status.ok()) {
 // 这里可以安全地读取状态机数据
 _callback(true, "OK");
 } else {
 _callback(false, status.error_str());
 }
 delete this;
 }

 private:
 std::function<void(bool, const std::string&)> _callback;
 };

 braft::Node* _node;
 };

 // 租约读（降低延迟，但一致性稍弱）
 class LeaseReader {
 public:
 LeaseReader(braft::Node* node, int64_t lease_timeout_ms = 9000)
 : _node(node), _lease_timeout_ms(lease_timeout_ms) {}

 void read(const std::function<void(bool, const std::string&)>& callback) {
 if (!_node->is_leader()) {
 callback(false, "Not leader");
 return;
 }

 int64_t now_ms = butil::gettimeofday_ms();
 if (now_ms - _last_heartbeat_ms < _lease_timeout_ms) {
 // 租约有效，直接读取
 callback(true, "OK");
 } else {
 // 租约过期，需要发送心跳确认 Leader 身份
 refresh_lease([callback](bool success) {
 callback(success, success ? "OK" : "Lease expired");
 });
 }
 }

 void on_heartbeat_success() {
 _last_heartbeat_ms = butil::gettimeofday_ms();
 }

 private:
 void refresh_lease(const std::function<void(bool)>& callback) {
 // 发送心跳以刷新租约
 HeartbeatClosure* done = new HeartbeatClosure([this, callback](bool success) {
 if (success) {
 _last_heartbeat_ms = butil::gettimeofday_ms();
 }
 callback(success);
 });

 // 这里需要向 Followers 发送心跳
 // 实际实现需要更复杂的逻辑
 braft::Task task;
 task.data = nullptr; // 空任务，仅用于确认 Leader 身份
 task.done = done;

 _node->apply(task);
 }

 class HeartbeatClosure : public braft::Closure {
 public:
 HeartbeatClosure(std::function<void(bool)> callback)
 : _callback(callback) {}

 void Run() override {
 _callback(status().ok());
 delete this;
 }

 private:
 std::function<void(bool)> _callback;
 };

 braft::Node* _node;
 int64_t _lease_timeout_ms;
 std::atomic<int64_t> _last_heartbeat_ms{0};
 };

 // 随机读（最终一致性，但延迟最低）
 class EventualReader {
 public:
 EventualReader(const std::vector<braft::Node*>& nodes)
 : _nodes(nodes) {}

 void read(const std::function<void(bool, const std::string&)>& callback) {
 if (_nodes.empty()) {
 callback(false, "No nodes available");
 return;
 }

 // 随机选择一个节点进行读取
 size_t index = rand() % _nodes.size();
 braft::Node* node = _nodes[index];

 // 直接读取状态机数据（不等待一致性确认）
 callback(true, "OK");
 }

 private:
 std::vector<braft::Node*> _nodes;
 };
};

四、最佳实践

4.1 错误处理和恢复

4.1.1 故障检测

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171


class FailureDetector {
public:
 enum FailureType {
 LEADER_ELECTION_TIMEOUT,
 LOG_REPLICATION_FAILURE,
 SNAPSHOT_FAILURE,
 CONFIGURATION_CHANGE_FAILURE,
 DISK_IO_ERROR,
 NETWORK_PARTITION
 };

 struct FailureEvent {
 FailureType type;
 std::string description;
 int64_t timestamp;
 butil::Status status;
 };

 class FailureHandler {
 public:
 virtual ~FailureHandler() = default;
 virtual void handle_failure(const FailureEvent& event) = 0;
 };

 FailureDetector(braft::Node* node) : _node(node) {
 _monitor_thread = std::thread([this] { monitor_loop(); });
 }

 ~FailureDetector() {
 _stopped.store(true);
 if (_monitor_thread.joinable()) {
 _monitor_thread.join();
 }
 }

 void add_handler(std::shared_ptr<FailureHandler> handler) {
 std::lock_guard<std::mutex> lock(_handlers_mutex);
 _handlers.push_back(handler);
 }

private:
 void monitor_loop() {
 int64_t last_leader_check = 0;
 int64_t last_health_check = 0;

 while (!_stopped.load()) {
 int64_t now = butil::gettimeofday_ms();

 // 检查 Leader 选举超时
 if (now - last_leader_check > 5000) { // 5秒检查一次
 check_leader_election();
 last_leader_check = now;
 }

 // 检查节点健康状态
 if (now - last_health_check > 10000) { // 10秒检查一次
 check_node_health();
 last_health_check = now;
 }

 bthread_usleep(1000 * 1000); // 1秒
 }
 }

 void check_leader_election() {
 if (!_node->is_leader()) {
 braft::PeerId leader = _node->leader_id();
 if (leader.is_empty()) {
 // 没有 Leader，可能选举超时
 notify_failure({
 LEADER_ELECTION_TIMEOUT,
 "No leader elected for extended period",
 butil::gettimeofday_ms(),
 butil::Status(ENOENT, "No leader")
 });
 }
 }
 }

 void check_node_health() {
 // 检查磁盘空间
 struct statvfs stat;
 if (statvfs(_data_dir.c_str(), &stat) == 0) {
 double free_space_ratio = static_cast<double>(stat.f_bavail) / stat.f_blocks;
 if (free_space_ratio < 0.1) { // 磁盘空间不足10%
 notify_failure({
 DISK_IO_ERROR,
 "Disk space critically low: " + std::to_string(free_space_ratio * 100) + "%",
 butil::gettimeofday_ms(),
 butil::Status(ENOSPC, "Disk space low")
 });
 }
 }
 }

 void notify_failure(const FailureEvent& event) {
 LOG(ERROR) << "Failure detected: " << event.description;

 std::lock_guard<std::mutex> lock(_handlers_mutex);
 for (auto& handler : _handlers) {
 if (auto h = handler.lock()) {
 h->handle_failure(event);
 }
 }
 }

 braft::Node* _node;
 std::string _data_dir;
 std::atomic<bool> _stopped{false};
 std::thread _monitor_thread;

 std::mutex _handlers_mutex;
 std::vector<std::weak_ptr<FailureHandler>> _handlers;
};

// 自动恢复处理器
class AutoRecoveryHandler : public FailureDetector::FailureHandler {
public:
 AutoRecoveryHandler(braft::Node* node) : _node(node) {}

 void handle_failure(const FailureDetector::FailureEvent& event) override {
 switch (event.type) {
 case FailureDetector::LEADER_ELECTION_TIMEOUT:
 handle_leader_election_timeout();
 break;

 case FailureDetector::LOG_REPLICATION_FAILURE:
 handle_log_replication_failure();
 break;

 case FailureDetector::SNAPSHOT_FAILURE:
 handle_snapshot_failure();
 break;

 case FailureDetector::DISK_IO_ERROR:
 handle_disk_error();
 break;

 default:
 LOG(WARNING) << "Unhandled failure type: " << event.type;
 }
 }

private:
 void handle_leader_election_timeout() {
 // 尝试手动触发选举
 LOG(INFO) << "Attempting to trigger leader election";
 _node->vote(0); // 发起选举
 }

 void handle_log_replication_failure() {
 // 检查是否需要创建快照来帮助落后的节点
 LOG(INFO) << "Creating snapshot to help log replication";
 _node->snapshot(nullptr);
 }

 void handle_snapshot_failure() {
 // 重试快照创建
 LOG(INFO) << "Retrying snapshot creation";
 bthread_usleep(5 * 1000 * 1000); // 等待5秒
 _node->snapshot(nullptr);
 }

 void handle_disk_error() {
 // 磁盘错误比较严重，可能需要人工干预
 LOG(FATAL) << "Disk error detected, manual intervention required";
 // 可以考虑优雅关闭节点
 }

 braft::Node* _node;
};

4.2 监控和观测

4.2.1 指标收集

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204


class RaftMetrics {
public:
 struct NodeMetrics {
 // 基础状态
 bool is_leader = false;
 int64_t current_term = 0;
 int64_t last_log_index = 0;
 int64_t commit_index = 0;

 // 性能指标
 int64_t apply_ops_per_second = 0;
 int64_t apply_latency_us = 0;
 int64_t log_replication_latency_us = 0;

 // 资源使用
 size_t log_storage_size = 0;
 size_t snapshot_size = 0;
 int32_t active_connections = 0;

 // 错误统计
 int64_t election_failures = 0;
 int64_t log_replication_failures = 0;
 int64_t snapshot_failures = 0;
 };

 RaftMetrics(braft::Node* node)
 : _node(node), _start_time(butil::gettimeofday_us()) {
 _metrics_thread = std::thread([this] { metrics_loop(); });
 }

 ~RaftMetrics() {
 _stopped.store(true);
 if (_metrics_thread.joinable()) {
 _metrics_thread.join();
 }
 }

 NodeMetrics get_metrics() const {
 std::lock_guard<std::mutex> lock(_metrics_mutex);
 return _current_metrics;
 }

 // 输出 Prometheus 格式的指标
 std::string to_prometheus() const {
 NodeMetrics metrics = get_metrics();
 std::ostringstream oss;

 oss << "# HELP raft_is_leader Whether this node is the leader\n";
 oss << "raft_is_leader " << (metrics.is_leader ? 1 : 0) << "\n";

 oss << "# HELP raft_current_term Current Raft term\n";
 oss << "raft_current_term " << metrics.current_term << "\n";

 oss << "# HELP raft_last_log_index Last log index\n";
 oss << "raft_last_log_index " << metrics.last_log_index << "\n";

 oss << "# HELP raft_commit_index Last committed log index\n";
 oss << "raft_commit_index " << metrics.commit_index << "\n";

 oss << "# HELP raft_apply_ops_per_second Operations applied per second\n";
 oss << "raft_apply_ops_per_second " << metrics.apply_ops_per_second << "\n";

 oss << "# HELP raft_apply_latency_microseconds Apply latency in microseconds\n";
 oss << "raft_apply_latency_microseconds " << metrics.apply_latency_us << "\n";

 oss << "# HELP raft_log_storage_size_bytes Log storage size in bytes\n";
 oss << "raft_log_storage_size_bytes " << metrics.log_storage_size << "\n";

 oss << "# HELP raft_election_failures_total Total election failures\n";
 oss << "raft_election_failures_total " << metrics.election_failures << "\n";

 return oss.str();
 }

 void record_apply_operation(int64_t latency_us) {
 std::lock_guard<std::mutex> lock(_metrics_mutex);
 _apply_operations.push_back({butil::gettimeofday_us(), latency_us});

 // 保留最近1分钟的数据
 int64_t cutoff = butil::gettimeofday_us() - 60 * 1000 * 1000;
 _apply_operations.erase(
 std::remove_if(_apply_operations.begin(), _apply_operations.end(),
 [cutoff](const ApplyRecord& r) { return r.timestamp < cutoff; }),
 _apply_operations.end()
 );
 }

 void record_election_failure() {
 std::lock_guard<std::mutex> lock(_metrics_mutex);
 _current_metrics.election_failures++;
 }

private:
 struct ApplyRecord {
 int64_t timestamp;
 int64_t latency_us;
 };

 void metrics_loop() {
 while (!_stopped.load()) {
 update_metrics();
 bthread_usleep(5 * 1000 * 1000); // 5秒更新一次
 }
 }

 void update_metrics() {
 std::lock_guard<std::mutex> lock(_metrics_mutex);

 // 更新基础状态
 _current_metrics.is_leader = _node->is_leader();

 // 计算 QPS
 int64_t now = butil::gettimeofday_us();
 int64_t window_start = now - 60 * 1000 * 1000; // 1分钟窗口

 auto valid_records = std::count_if(
 _apply_operations.begin(), _apply_operations.end(),
 [window_start](const ApplyRecord& r) { return r.timestamp >= window_start; }
 );

 _current_metrics.apply_ops_per_second = valid_records / 60;

 // 计算平均延迟
 if (!_apply_operations.empty()) {
 int64_t total_latency = 0;
 for (const auto& record : _apply_operations) {
 if (record.timestamp >= window_start) {
 total_latency += record.latency_us;
 }
 }
 _current_metrics.apply_latency_us =
 valid_records > 0 ? total_latency / valid_records : 0;
 }

 // 更新存储大小（需要从实际存储获取）
 update_storage_metrics();
 }

 void update_storage_metrics() {
 // 这里需要从实际存储后端获取大小信息
 // 具体实现依赖于存储配置
 }

 braft::Node* _node;
 int64_t _start_time;

 mutable std::mutex _metrics_mutex;
 NodeMetrics _current_metrics;
 std::vector<ApplyRecord> _apply_operations;

 std::atomic<bool> _stopped{false};
 std::thread _metrics_thread;
};

// HTTP 指标服务器
class MetricsServer {
public:
 MetricsServer(RaftMetrics* metrics, int port)
 : _metrics(metrics), _port(port) {}

 int start() {
 brpc::ServerOptions options;
 if (_server.AddService(&_service, brpc::SERVER_DOESNT_OWN_SERVICE) != 0) {
 LOG(ERROR) << "Failed to add metrics service";
 return -1;
 }

 if (_server.Start(_port, &options) != 0) {
 LOG(ERROR) << "Failed to start metrics server on port " << _port;
 return -1;
 }

 LOG(INFO) << "Metrics server started on port " << _port;
 return 0;
 }

private:
 class MetricsService : public brpc::HttpService {
 public:
 MetricsService(RaftMetrics* metrics) : _metrics(metrics) {}

 void default_method(google::protobuf::RpcController* cntl_base,
 const brpc::HttpRequest*,
 brpc::HttpResponse*,
 google::protobuf::Closure* done) override {
 brpc::ClosureGuard done_guard(done);
 brpc::Controller* cntl = static_cast<brpc::Controller*>(cntl_base);

 cntl->http_response().set_content_type("text/plain");
 cntl->http_response().set_status_code(brpc::HTTP_STATUS_OK);

 std::string metrics = _metrics->to_prometheus();
 cntl->response_attachment().append(metrics);
 }

 private:
 RaftMetrics* _metrics;
 };

 RaftMetrics* _metrics;
 int _port;
 brpc::Server _server;
 MetricsService _service{_metrics};
};

4.3 生产环境部署

4.3.1 配置优化

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178


class ProductionConfig {
public:
 // 生产环境推荐配置
 static braft::NodeOptions get_production_options(const std::string& data_dir) {
 braft::NodeOptions options;

 // 存储配置
 options.log_uri = data_dir + "/log";
 options.raft_meta_uri = data_dir + "/raft_meta";
 options.snapshot_uri = data_dir + "/snapshot";

 // 选举配置
 options.election_timeout_ms = 1000; // 1秒选举超时
 options.heartbeat_interval_ms = 250; // 250ms心跳间隔
 options.apply_batch = 32; // 批量应用32个操作

 // 快照配置
 options.snapshot_interval_s = 3600; // 1小时快照间隔
 options.snapshot_throttle = nullptr; // 不限制快照速度

 // 日志配置
 options.max_byte_count_per_rpc = 128 * 1024; // 128KB per RPC
 options.max_entries_per_rpc = 1024; // 1024 entries per RPC
 options.max_body_size = 512 * 1024; // 512KB max body

 // 性能优化
 options.sync = true; // 同步写入（保证持久性）
 options.sync_meta = true; // 同步写入元数据

 // 启用内置服务
 options.disable_cli = false; // 启用 CLI

 return options;
 }

 // 高性能配置（牺牲一些持久性保证）
 static braft::NodeOptions get_high_performance_options(const std::string& data_dir) {
 auto options = get_production_options(data_dir);

 // 减少同步写入
 options.sync = false; // 异步写入（性能优先）
 options.sync_meta = false;

 // 增加批量大小
 options.apply_batch = 128;
 options.max_entries_per_rpc = 4096;

 // 减少选举超时
 options.election_timeout_ms = 500;
 options.heartbeat_interval_ms = 100;

 return options;
 }

 // 高可用配置（最大化可靠性）
 static braft::NodeOptions get_high_availability_options(const std::string& data_dir) {
 auto options = get_production_options(data_dir);

 // 更保守的同步设置
 options.sync = true;
 options.sync_meta = true;

 // 更频繁的快照
 options.snapshot_interval_s = 1800; // 30分钟快照

 // 更小的批量大小（更快的故障检测）
 options.apply_batch = 16;

 // 更长的选举超时（避免网络抖动导致的选举）
 options.election_timeout_ms = 2000;
 options.heartbeat_interval_ms = 500;

 return options;
 }
};

// 集群部署辅助工具
class ClusterDeployer {
public:
 struct NodeConfig {
 std::string node_id;
 std::string listen_addr;
 std::string raft_addr;
 std::string data_dir;
 };

 // 部署集群
 static std::vector<CounterServer*> deploy_cluster(
 const std::vector<NodeConfig>& configs) {

 std::vector<CounterServer*> servers;

 // 构建初始 peers 列表
 std::string peers;
 for (size_t i = 0; i < configs.size(); ++i) {
 if (i > 0) peers += ",";
 peers += configs[i].raft_addr;
 }

 // 启动所有节点
 for (const auto& config : configs) {
 CounterServer* server = new CounterServer();

 // 创建数据目录
 system(("mkdir -p " + config.data_dir).c_str());

 if (server->start(config.listen_addr, config.raft_addr,
 peers, config.data_dir) != 0) {
 LOG(ERROR) << "Failed to start server " << config.node_id;
 delete server;
 continue;
 }

 servers.push_back(server);
 LOG(INFO) << "Started server " << config.node_id
 << " at " << config.listen_addr;
 }

 return servers;
 }

 // 滚动升级
 static bool rolling_upgrade(std::vector<CounterServer*>& servers,
 const std::string& new_binary_path) {
 for (size_t i = 0; i < servers.size(); ++i) {
 LOG(INFO) << "Upgrading server " << i;

 // 1. 停止当前服务器
 delete servers[i];

 // 2. 替换二进制文件
 std::string backup_cmd = "cp " + new_binary_path + " ./server_" + std::to_string(i);
 if (system(backup_cmd.c_str()) != 0) {
 LOG(ERROR) << "Failed to copy new binary";
 return false;
 }

 // 3. 启动新版本
 // 这里需要重新启动服务器实例
 // 实际实现需要保存启动参数

 // 4. 等待节点重新加入集群
 bthread_usleep(10 * 1000 * 1000); // 等待10秒

 LOG(INFO) << "Server " << i << " upgraded successfully";
 }

 return true;
 }

 // 集群健康检查
 static bool health_check(const std::vector<CounterServer*>& servers) {
 int leader_count = 0;
 int healthy_count = 0;

 for (const auto* server : servers) {
 if (server->is_leader()) {
 leader_count++;
 }

 // 简单的健康检查
 try {
 server->get_counter_local();
 healthy_count++;
 } catch (...) {
 LOG(WARNING) << "Server health check failed";
 }
 }

 bool healthy = (leader_count == 1) &&
 (healthy_count >= (servers.size() + 1) / 2);

 LOG(INFO) << "Cluster health: " << healthy_count << "/" << servers.size()
 << " healthy, " << leader_count << " leaders";

 return healthy;
 }
};

五、故障排查和调试

5.1 常见问题诊断

5.1.1 选举问题

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79


class ElectionDiagnostics {
public:
 static void diagnose_election_issues(braft::Node* node) {
 LOG(INFO) << "=== Election Diagnostics ===";

 // 检查当前状态
 bool is_leader = node->is_leader();
 braft::PeerId leader_id = node->leader_id();

 LOG(INFO) << "Current state - Is leader: " << is_leader
 << ", Leader: " << leader_id;

 // 检查配置
 braft::Configuration conf;
 node->list_peers(&conf);

 LOG(INFO) << "Cluster configuration:";
 for (auto it = conf.begin(); it != conf.end(); ++it) {
 LOG(INFO) << " Peer: " << *it;
 }

 // 检查网络连通性
 check_network_connectivity(conf);

 // 检查时钟同步
 check_clock_sync();

 // 检查磁盘IO
 check_disk_io();
 }

private:
 static void check_network_connectivity(const braft::Configuration& conf) {
 LOG(INFO) << "Checking network connectivity...";

 for (auto it = conf.begin(); it != conf.end(); ++it) {
 std::string addr = it->addr.ip.to_string() + ":" +
 std::to_string(it->addr.port);

 // 简单的连通性检查
 brpc::Channel channel;
 brpc::ChannelOptions options;
 options.timeout_ms = 1000;

 if (channel.Init(addr.c_str(), &options) == 0) {
 LOG(INFO) << " " << addr << ": OK";
 } else {
 LOG(ERROR) << " " << addr << ": FAILED";
 }
 }
 }

 static void check_clock_sync() {
 // 检查系统时钟是否同步
 LOG(INFO) << "System time: " << time(nullptr);
 // 在生产环境中，应该检查 NTP 同步状态
 }

 static void check_disk_io() {
 // 检查磁盘 IO 性能
 auto start = std::chrono::high_resolution_clock::now();

 std::string test_file = "/tmp/braft_io_test";
 std::ofstream file(test_file);
 if (file.is_open()) {
 file << "test data";
 file.flush();
 file.close();
 }

 auto end = std::chrono::high_resolution_clock::now();
 auto duration = std::chrono::duration_cast<std::chrono::microseconds>(
 end - start).count();

 LOG(INFO) << "Disk write latency: " << duration << "us";

 unlink(test_file.c_str());
 }
};

5.1.2 性能问题诊断

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101


class PerformanceDiagnostics {
public:
 struct PerformanceReport {
 double apply_latency_p50_ms = 0;
 double apply_latency_p99_ms = 0;
 double throughput_ops_per_sec = 0;
 double log_replication_latency_ms = 0;
 size_t pending_tasks = 0;
 double cpu_usage_percent = 0;
 double memory_usage_mb = 0;
 };

 static PerformanceReport generate_report(braft::Node* node) {
 PerformanceReport report;

 // 收集延迟统计
 collect_latency_stats(report);

 // 收集吞吐量统计
 collect_throughput_stats(report);

 // 收集系统资源使用
 collect_system_stats(report);

 // 收集 Raft 特定指标
 collect_raft_stats(node, report);

 return report;
 }

 static void print_report(const PerformanceReport& report) {
 LOG(INFO) << "=== Performance Report ===";
 LOG(INFO) << "Apply Latency P50: " << report.apply_latency_p50_ms << "ms";
 LOG(INFO) << "Apply Latency P99: " << report.apply_latency_p99_ms << "ms";
 LOG(INFO) << "Throughput: " << report.throughput_ops_per_sec << " ops/sec";
 LOG(INFO) << "Log Replication Latency: " << report.log_replication_latency_ms << "ms";
 LOG(INFO) << "Pending Tasks: " << report.pending_tasks;
 LOG(INFO) << "CPU Usage: " << report.cpu_usage_percent << "%";
 LOG(INFO) << "Memory Usage: " << report.memory_usage_mb << "MB";
 LOG(INFO) << "========================";
 }

 static std::vector<std::string> analyze_bottlenecks(const PerformanceReport& report) {
 std::vector<std::string> issues;

 if (report.apply_latency_p99_ms > 100) {
 issues.push_back("High apply latency detected - consider optimizing state machine");
 }

 if (report.log_replication_latency_ms > 50) {
 issues.push_back("High log replication latency - check network or reduce batch size");
 }

 if (report.pending_tasks > 1000) {
 issues.push_back("High number of pending tasks - system may be overloaded");
 }

 if (report.cpu_usage_percent > 80) {
 issues.push_back("High CPU usage - consider scaling or optimizing");
 }

 if (report.memory_usage_mb > 1024) {
 issues.push_back("High memory usage - check for memory leaks");
 }

 if (report.throughput_ops_per_sec < 100) {
 issues.push_back("Low throughput - investigate performance bottlenecks");
 }

 return issues;
 }

private:
 static void collect_latency_stats(PerformanceReport& report) {
 // 这里需要从实际的延迟统计中收集数据
 // 示例数据
 report.apply_latency_p50_ms = 5.2;
 report.apply_latency_p99_ms = 25.8;
 }

 static void collect_throughput_stats(PerformanceReport& report) {
 // 从指标收集器获取吞吐量数据
 report.throughput_ops_per_sec = 1500;
 }

 static void collect_system_stats(PerformanceReport& report) {
 // 获取系统资源使用情况
 std::ifstream stat_file("/proc/stat");
 std::ifstream mem_file("/proc/meminfo");

 // 简化的系统统计收集
 report.cpu_usage_percent = 35.2;
 report.memory_usage_mb = 512.7;
 }

 static void collect_raft_stats(braft::Node* node, PerformanceReport& report) {
 // 收集 Raft 相关统计
 report.log_replication_latency_ms = 12.3;
 report.pending_tasks = 45;
 }
};

5.2 调试工具

5.2.1 状态查看器

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207


class StateViewer {
public:
 static void dump_node_state(braft::Node* node, std::ostream& os) {
 os << "=== Node State Dump ===" << std::endl;

 // 基本信息
 os << "Node ID: " << node->node_id() << std::endl;
 os << "Is Leader: " << (node->is_leader() ? "Yes" : "No") << std::endl;
 os << "Leader ID: " << node->leader_id() << std::endl;

 // 配置信息
 braft::Configuration conf;
 node->list_peers(&conf);
 os << "Cluster Members:" << std::endl;
 for (auto it = conf.begin(); it != conf.end(); ++it) {
 os << " " << *it << std::endl;
 }

 // 日志信息
 dump_log_info(node, os);

 // 状态机信息
 dump_state_machine_info(node, os);

 os << "======================" << std::endl;
 }

 static void dump_log_info(braft::Node* node, std::ostream& os) {
 os << "Log Information:" << std::endl;
 // 这里需要从 braft 内部获取日志信息
 // 由于接口限制，这里只能输出可访问的信息
 os << " Last Log Index: " << "N/A" << std::endl;
 os << " Commit Index: " << "N/A" << std::endl;
 os << " Last Applied: " << "N/A" << std::endl;
 }

 static void dump_state_machine_info(braft::Node* node, std::ostream& os) {
 os << "State Machine Information:" << std::endl;
 // 这里需要从状态机获取信息
 os << " Implementation specific data would go here" << std::endl;
 }

 // 导出配置历史
 static void dump_config_history(braft::Node* node, std::ostream& os) {
 os << "Configuration Change History:" << std::endl;
 // 配置变更历史需要自己维护
 os << " Implementation needed" << std::endl;
 }
};

// 交互式调试器
class InteractiveDebugger {
public:
 InteractiveDebugger(braft::Node* node) : _node(node) {}

 void start_interactive_session() {
 std::cout << "braft Interactive Debugger" << std::endl;
 std::cout << "Type 'help' for available commands" << std::endl;

 std::string line;
 while (std::getline(std::cin, line)) {
 if (line == "quit" || line == "exit") {
 break;
 }

 process_command(line);
 }
 }

private:
 void process_command(const std::string& cmd) {
 std::istringstream iss(cmd);
 std::string command;
 iss >> command;

 if (command == "help") {
 show_help();
 } else if (command == "status") {
 show_status();
 } else if (command == "peers") {
 show_peers();
 } else if (command == "logs") {
 show_logs();
 } else if (command == "snapshot") {
 create_snapshot();
 } else if (command == "transfer") {
 std::string target;
 iss >> target;
 transfer_leadership(target);
 } else if (command == "add_peer") {
 std::string peer;
 iss >> peer;
 add_peer(peer);
 } else if (command == "remove_peer") {
 std::string peer;
 iss >> peer;
 remove_peer(peer);
 } else {
 std::cout << "Unknown command: " << command << std::endl;
 }
 }

 void show_help() {
 std::cout << "Available commands:" << std::endl;
 std::cout << " status - Show node status" << std::endl;
 std::cout << " peers - Show cluster members" << std::endl;
 std::cout << " logs - Show log information" << std::endl;
 std::cout << " snapshot - Create snapshot" << std::endl;
 std::cout << " transfer <peer> - Transfer leadership" << std::endl;
 std::cout << " add_peer <addr> - Add cluster member" << std::endl;
 std::cout << " remove_peer <addr> - Remove cluster member" << std::endl;
 std::cout << " quit/exit - Exit debugger" << std::endl;
 }

 void show_status() {
 StateViewer::dump_node_state(_node, std::cout);
 }

 void show_peers() {
 braft::Configuration conf;
 _node->list_peers(&conf);

 std::cout << "Cluster Members:" << std::endl;
 for (auto it = conf.begin(); it != conf.end(); ++it) {
 std::cout << " " << *it;
 if (*it == _node->leader_id()) {
 std::cout << " (Leader)";
 }
 std::cout << std::endl;
 }
 }

 void show_logs() {
 std::cout << "Log information display not implemented" << std::endl;
 }

 void create_snapshot() {
 std::cout << "Creating snapshot..." << std::endl;
 _node->snapshot(nullptr);
 std::cout << "Snapshot creation initiated" << std::endl;
 }

 void transfer_leadership(const std::string& target) {
 if (target.empty()) {
 std::cout << "Usage: transfer <peer_address>" << std::endl;
 return;
 }

 braft::PeerId target_peer(target);
 std::cout << "Transferring leadership to " << target << "..." << std::endl;

 int ret = _node->transfer_leadership_to(target_peer);
 if (ret == 0) {
 std::cout << "Leadership transfer initiated" << std::endl;
 } else {
 std::cout << "Failed to transfer leadership: " << ret << std::endl;
 }
 }

 void add_peer(const std::string& peer_addr) {
 if (peer_addr.empty()) {
 std::cout << "Usage: add_peer <peer_address>" << std::endl;
 return;
 }

 std::cout << "Adding peer " << peer_addr << "..." << std::endl;

 class AddPeerClosure : public braft::Closure {
 public:
 void Run() override {
 if (status().ok()) {
 std::cout << "Peer added successfully" << std::endl;
 } else {
 std::cout << "Failed to add peer: " << status() << std::endl;
 }
 delete this;
 }
 };

 ConfigurationManager::add_peer(_node, peer_addr, new AddPeerClosure());
 }

 void remove_peer(const std::string& peer_addr) {
 if (peer_addr.empty()) {
 std::cout << "Usage: remove_peer <peer_address>" << std::endl;
 return;
 }

 std::cout << "Removing peer " << peer_addr << "..." << std::endl;

 class RemovePeerClosure : public braft::Closure {
 public:
 void Run() override {
 if (status().ok()) {
 std::cout << "Peer removed successfully" << std::endl;
 } else {
 std::cout << "Failed to remove peer: " << status() << std::endl;
 }
 delete this;
 }
 };

 ConfigurationManager::remove_peer(_node, peer_addr, new RemovePeerClosure());
 }

 braft::Node* _node;
};

六、学习资源和总结

6.1 深入学习路径

基础阶段（1-2周）
- 理解 Raft 算法原理
- 完成基础的计数器示例
- 掌握状态机接口
进阶阶段（2-4周）
- 实现复杂的状态机（如 KV 存储）
- 掌握快照机制
- 理解配置变更
高级阶段（1-2月）
- 性能优化和调优
- 生产环境部署
- 故障处理和恢复
专家阶段
- 源码阅读和定制
- 与其他系统集成
- 贡献开源社区

6.2 最佳实践总结

设计原则
- 状态机操作要幂等
- 日志条目要自包含
- 避免阻塞操作
性能优化
- 使用批量提交
- 合理设置快照间隔
- 优化网络和磁盘 I/O
可靠性保证
- 确保数据持久化
- 监控集群健康状态
- 准备故障恢复方案

6.3 常见陷阱

状态机非幂等：确保重复应用相同日志条目不会产生副作用
阻塞状态机：避免在状态机中执行耗时操作
忽略错误处理：必须正确处理各种异常情况
配置变更不当：配置变更要谨慎，避免脑裂

6.4 参考资源

braft 作为生产级的 Raft 实现，为构建分布式一致性系统提供了强大的基础。通过本指南的学习，相信你已经掌握了使用 braft 构建可靠分布式系统的核心技能。记住，分布式系统的精髓在于处理各种异常情况，只有在实践中不断积累经验，才能真正掌握这门艺术。

]]>

brpc 深度学习指南：从入门到高级

lesshash@blog.lesshash.com/ (lesshash) — Thu, 18 Sep 2025 14:55:00 +0800

一、为什么选择 brpc？

1.1 使用背景

在现代互联网架构中，微服务已经成为主流架构模式。当系统被拆分成多个服务后，服务间的高效通信变得至关重要。传统的 HTTP REST API 虽然简单，但在高并发、低延迟场景下性能不足。这时候就需要一个高性能的 RPC 框架。

brpc（better RPC）是百度开源并贡献给 Apache 基金会的工业级 RPC 框架，广泛应用于百度内部的搜索、存储、机器学习、广告、推荐等高性能系统。

1.2 核心优势

性能优势

极高的吞吐量：单连接可达 800MB/s 吞吐量，多连接模式下可达 2.3GB/s
超低延迟：最小化锁竞争，即使在 50万+ QPS 下也几乎看不到锁竞争
优于 gRPC：同等条件下，brpc 的 QPS 比 gRPC 高约 10000，且在大包（>8KB）场景下性能优势更明显

架构优势

无 IO/业务线程分离：brpc 智能地将 IO 和业务处理组合，实现更好的并发和效率
动态线程管理：根据负载自动调整线程数，每个请求创建一个 bthread，处理完自动结束
完全并行化：不同客户端的请求读取和解析完全并行化

易用性优势

协议自动识别：服务器自动检测支持的协议，一个端口可接受多种协议
丰富的协议支持：支持 HTTP/HTTPS、H2/gRPC、Redis、Memcached 等多种协议
完善的调试工具：内置强大的性能分析和调试工具

二、快速入门

2.1 环境准备

Ubuntu/Debian 系统

1
2
3
4
5
6
7
8


# 安装基础依赖
sudo apt-get install -y git g++ make libssl-dev libgflags-dev \
 libprotobuf-dev libprotoc-dev protobuf-compiler libleveldb-dev

# 安装测试框架（可选）
sudo apt-get install -y cmake libgtest-dev
cd /usr/src/gtest && sudo cmake . && sudo make
sudo mv lib/libgtest* /usr/lib/ && cd -

CentOS 系统

1
2
3
4
5
6


# 安装 EPEL
sudo yum install epel-release

# 安装依赖
sudo yum install git gcc-c++ make openssl-devel gflags-devel \
 protobuf-devel protobuf-compiler leveldb-devel

2.2 编译安装 brpc

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# 克隆代码
git clone https://github.com/apache/brpc.git
cd brpc

# 编译
sh config_brpc.sh --headers=/usr/include --libs=/usr/lib
make -j8

# 运行示例
cd example/echo_c++
make
./echo_server &
./echo_client

2.3 第一个 brpc 程序

Step 1: 定义服务接口（echo.proto）

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


syntax = "proto2";
package example;

option cc_generic_services = true;

message EchoRequest {
 required string message = 1;
}

message EchoResponse {
 required string message = 1;
}

service EchoService {
 rpc Echo(EchoRequest) returns (EchoResponse);
}

Step 2: 生成代码

1

protoc --cpp_out=. echo.proto

Step 3: 实现服务端

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


#include <brpc/server.h>
#include "echo.pb.h"

// 实现服务
class EchoServiceImpl : public example::EchoService {
public:
 void Echo(google::protobuf::RpcController* cntl_base,
 const example::EchoRequest* request,
 example::EchoResponse* response,
 google::protobuf::Closure* done) {
 brpc::ClosureGuard done_guard(done);

 // 业务逻辑
 response->set_message(request->message());

 LOG(INFO) << "Received: " << request->message();
 }
};

int main(int argc, char* argv[]) {
 // 创建服务器
 brpc::Server server;

 // 添加服务
 EchoServiceImpl echo_service;
 if (server.AddService(&echo_service,
 brpc::SERVER_DOESNT_OWN_SERVICE) != 0) {
 LOG(ERROR) << "Fail to add service";
 return -1;
 }

 // 启动服务器
 brpc::ServerOptions options;
 options.num_threads = 8; // 工作线程数
 if (server.Start(8000, &options) != 0) {
 LOG(ERROR) << "Fail to start server";
 return -1;
 }

 // 等待退出
 server.RunUntilAskedToQuit();
 return 0;
}

Step 4: 实现客户端

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


#include <brpc/channel.h>
#include "echo.pb.h"

int main(int argc, char* argv[]) {
 // 创建 Channel
 brpc::Channel channel;
 brpc::ChannelOptions options;
 options.protocol = brpc::PROTOCOL_BAIDU_STD;
 options.timeout_ms = 100;
 options.max_retry = 3;

 if (channel.Init("127.0.0.1:8000", &options) != 0) {
 LOG(ERROR) << "Fail to initialize channel";
 return -1;
 }

 // 创建服务 stub
 example::EchoService_Stub stub(&channel);

 // 发送请求
 example::EchoRequest request;
 example::EchoResponse response;
 brpc::Controller cntl;

 request.set_message("Hello brpc!");
 stub.Echo(&cntl, &request, &response, NULL);

 if (!cntl.Failed()) {
 LOG(INFO) << "Response: " << response.message();
 } else {
 LOG(ERROR) << cntl.ErrorText();
 }

 return 0;
}

三、核心概念详解

3.1 bthread（协程）- brpc 的核心

3.1.1 什么是 bthread

bthread 是 brpc 实现的 M:N 线程库，是 brpc 高性能的关键。与传统的 pthread 不同：

轻量级：创建和切换开销极小
高并发：单机可支持百万级 bthread
同步编程：用同步方式写异步代码

3.1.2 基础 bthread 使用

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


#include <bthread/bthread.h>

// 基础 bthread 创建
void* worker_function(void* arg) {
 LOG(INFO) << "Running in bthread " << bthread_self();

 // 模拟工作
 bthread_usleep(1000); // 睡眠 1ms，不会阻塞 pthread

 return NULL;
}

int main() {
 bthread_t tid;

 // 创建 bthread
 if (bthread_start_urgent(&tid, NULL, worker_function, NULL) != 0) {
 LOG(ERROR) << "Fail to create bthread";
 return -1;
 }

 // 等待 bthread 结束
 bthread_join(tid, NULL);
 return 0;
}

3.1.3 bthread 批量创建示例

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46


#include <vector>
#include <atomic>

std::atomic<int> counter{0};

void* count_task(void* arg) {
 int* task_id = (int*)arg;

 // 模拟计算任务
 for (int i = 0; i < 1000; ++i) {
 counter.fetch_add(1);
 }

 LOG(INFO) << "Task " << *task_id << " completed by bthread "
 << bthread_self();
 return NULL;
}

void BatchCreateBthreads() {
 const int THREAD_COUNT = 10000; // 创建 1万个 bthread
 std::vector<bthread_t> threads(THREAD_COUNT);
 std::vector<int> task_ids(THREAD_COUNT);

 auto start = std::chrono::high_resolution_clock::now();

 // 批量创建 bthread
 for (int i = 0; i < THREAD_COUNT; ++i) {
 task_ids[i] = i;
 if (bthread_start_background(&threads[i], NULL,
 count_task, &task_ids[i]) != 0) {
 LOG(ERROR) << "Fail to create bthread " << i;
 }
 }

 // 等待所有 bthread 完成
 for (int i = 0; i < THREAD_COUNT; ++i) {
 bthread_join(threads[i], NULL);
 }

 auto end = std::chrono::high_resolution_clock::now();
 auto duration = std::chrono::duration_cast<std::chrono::milliseconds>(
 end - start).count();

 LOG(INFO) << "Created " << THREAD_COUNT << " bthreads in "
 << duration << "ms, counter=" << counter.load();
}

3.1.4 bthread 同步原语

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66


#include <bthread/mutex.h>
#include <bthread/condition_variable.h>

// bthread 专用的同步原语
class BthreadSafeQueue {
private:
 std::queue<int> queue_;
 bthread::Mutex mutex_;
 bthread::ConditionVariable cv_;
 bool stopped_ = false;

public:
 void Push(int value) {
 std::unique_lock<bthread::Mutex> lock(mutex_);
 queue_.push(value);
 cv_.notify_one();
 }

 bool Pop(int* value) {
 std::unique_lock<bthread::Mutex> lock(mutex_);

 // 等待有数据或停止信号
 cv_.wait(lock, [this] {
 return !queue_.empty() || stopped_;
 });

 if (stopped_ && queue_.empty()) {
 return false;
 }

 *value = queue_.front();
 queue_.pop();
 return true;
 }

 void Stop() {
 std::unique_lock<bthread::Mutex> lock(mutex_);
 stopped_ = true;
 cv_.notify_all();
 }
};

// 生产者-消费者示例
void* producer(void* arg) {
 BthreadSafeQueue* queue = (BthreadSafeQueue*)arg;

 for (int i = 0; i < 100; ++i) {
 queue->Push(i);
 bthread_usleep(100); // 模拟生产耗时
 LOG(INFO) << "Produced: " << i;
 }

 return NULL;
}

void* consumer(void* arg) {
 BthreadSafeQueue* queue = (BthreadSafeQueue*)arg;
 int value;

 while (queue->Pop(&value)) {
 LOG(INFO) << "Consumed: " << value;
 bthread_usleep(200); // 模拟消费耗时
 }

 return NULL;
}

3.1.5 bthread 本地存储

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


#include <bthread/key.h>

// bthread 本地存储（类似 thread_local）
bthread_key_t g_key;

void InitBthreadLocal() {
 // 创建 bthread 本地存储键
 if (bthread_key_create(&g_key, NULL) != 0) {
 LOG(ERROR) << "Fail to create bthread key";
 }
}

void* worker_with_local_storage(void* arg) {
 int* worker_id = (int*)arg;

 // 设置 bthread 本地数据
 bthread_setspecific(g_key, worker_id);

 // 模拟多层函数调用
 ProcessData();

 return NULL;
}

void ProcessData() {
 // 在任何函数中都可以获取 bthread 本地数据
 int* worker_id = (int*)bthread_getspecific(g_key);
 LOG(INFO) << "Processing data in worker " << *worker_id;

 // 继续调用其他函数
 CallSubFunction();
}

void CallSubFunction() {
 int* worker_id = (int*)bthread_getspecific(g_key);
 LOG(INFO) << "SubFunction called by worker " << *worker_id;
}

3.1.6 bthread 定时器

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52


#include <bthread/timer.h>

class BthreadTimer {
private:
 bthread_timer_t timer_id_;
 std::atomic<bool> running_{false};

public:
 // 定时器回调函数
 static void TimerCallback(void* arg) {
 BthreadTimer* timer = (BthreadTimer*)arg;
 LOG(INFO) << "Timer triggered at " << time(NULL);

 // 可以在这里执行任何异步任务
 timer->ProcessPeriodicTask();
 }

 void Start(int interval_ms) {
 if (running_.exchange(true)) {
 return; // 已经在运行
 }

 // 创建周期性定时器
 if (bthread_timer_add(&timer_id_,
 butil::milliseconds_from_now(interval_ms),
 TimerCallback, this) != 0) {
 LOG(ERROR) << "Fail to add timer";
 running_ = false;
 }
 }

 void Stop() {
 if (!running_.exchange(false)) {
 return; // 已经停止
 }

 bthread_timer_del(timer_id_);
 }

private:
 void ProcessPeriodicTask() {
 // 执行周期性任务
 LOG(INFO) << "Executing periodic task in bthread " << bthread_self();

 // 如果还在运行，则安排下次执行
 if (running_) {
 bthread_timer_add(&timer_id_,
 butil::milliseconds_from_now(1000),
 TimerCallback, this);
 }
 }
};

3.1.7 bthread 执行队列

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53


#include <bthread/execution_queue.h>

// 高性能的单线程执行队列
class TaskProcessor {
private:
 bthread::ExecutionQueueId<int> queue_id_;

public:
 // 任务处理函数
 static int ProcessTask(void* meta, bthread::TaskIterator<int>& iter) {
 TaskProcessor* processor = (TaskProcessor*)meta;

 // 批量处理任务
 for (; iter; ++iter) {
 processor->HandleSingleTask(*iter);
 }

 return 0; // 返回0表示继续处理
 }

 bool Init() {
 bthread::ExecutionQueueOptions options;

 if (bthread::execution_queue_start(&queue_id_, &options,
 ProcessTask, this) != 0) {
 LOG(ERROR) << "Fail to start execution queue";
 return false;
 }

 return true;
 }

 void SubmitTask(int task_data) {
 // 提交任务到队列（线程安全）
 if (bthread::execution_queue_execute(queue_id_, task_data) != 0) {
 LOG(ERROR) << "Fail to submit task";
 }
 }

 void Shutdown() {
 bthread::execution_queue_stop(queue_id_);
 bthread::execution_queue_join(queue_id_);
 }

private:
 void HandleSingleTask(int task_data) {
 LOG(INFO) << "Processing task: " << task_data
 << " in bthread " << bthread_self();

 // 模拟任务处理
 bthread_usleep(100);
 }
};

3.1.8 bthread 与 pthread 交互

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


// bthread 调用阻塞的 pthread 操作
class BthreadPthreadBridge {
public:
 // 在 bthread 中调用可能阻塞的操作
 static void CallBlockingOperation() {
 // 方法1：直接调用（会阻塞当前 pthread worker）
 // sleep(1); // 不推荐，会阻塞整个 worker 线程

 // 方法2：使用 bthread_usleep（推荐）
 bthread_usleep(1000000); // 睡眠1秒，只阻塞当前 bthread

 // 方法3：对于无法替换的阻塞调用，移到 pthread 执行
 std::future<int> result = std::async(std::launch::async, []() {
 sleep(1); // 在独立的 pthread 中执行
 return 42;
 });

 // 在 bthread 中等待结果
 int value = result.get();
 LOG(INFO) << "Got result: " << value;
 }

 // pthread 中启动 bthread
 static void PthreadStartBthread() {
 std::thread([&] {
 LOG(INFO) << "In pthread: " << std::this_thread::get_id();

 // 在 pthread 中启动 bthread
 bthread_t tid;
 bthread_start_urgent(&tid, NULL, [](void*) -> void* {
 LOG(INFO) << "In bthread: " << bthread_self();
 return NULL;
 }, NULL);

 bthread_join(tid, NULL);
 }).join();
 }
};

3.1.9 bthread 性能测试

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89


#include <chrono>

class BthreadPerformanceTest {
public:
 // 测试 bthread vs pthread 创建性能
 static void CompareCreationPerformance() {
 const int THREAD_COUNT = 10000;

 // 测试 bthread 创建性能
 auto start = std::chrono::high_resolution_clock::now();

 std::vector<bthread_t> bthreads(THREAD_COUNT);
 for (int i = 0; i < THREAD_COUNT; ++i) {
 bthread_start_background(&bthreads[i], NULL, [](void*) -> void* {
 bthread_usleep(1000); // 1ms
 return NULL;
 }, NULL);
 }

 for (int i = 0; i < THREAD_COUNT; ++i) {
 bthread_join(bthreads[i], NULL);
 }

 auto bthread_duration = std::chrono::duration_cast<std::chrono::milliseconds>(
 std::chrono::high_resolution_clock::now() - start).count();

 // 测试 pthread 创建性能
 start = std::chrono::high_resolution_clock::now();

 std::vector<std::thread> pthreads;
 pthreads.reserve(THREAD_COUNT);

 for (int i = 0; i < THREAD_COUNT; ++i) {
 pthreads.emplace_back([]() {
 std::this_thread::sleep_for(std::chrono::milliseconds(1));
 });
 }

 for (auto& t : pthreads) {
 t.join();
 }

 auto pthread_duration = std::chrono::duration_cast<std::chrono::milliseconds>(
 std::chrono::high_resolution_clock::now() - start).count();

 LOG(INFO) << "bthread creation: " << bthread_duration << "ms";
 LOG(INFO) << "pthread creation: " << pthread_duration << "ms";
 LOG(INFO) << "bthread is " << (pthread_duration / bthread_duration)
 << "x faster";
 }

 // 测试 bthread 上下文切换性能
 static void TestContextSwitch() {
 const int SWITCH_COUNT = 1000000;
 bthread::Mutex mutex;

 auto start = std::chrono::high_resolution_clock::now();

 bthread_t tid1, tid2;

 bthread_start_urgent(&tid1, NULL, [](void* arg) -> void* {
 bthread::Mutex* m = (bthread::Mutex*)arg;
 for (int i = 0; i < SWITCH_COUNT / 2; ++i) {
 std::unique_lock<bthread::Mutex> lock(*m);
 bthread_yield(); // 主动让出CPU
 }
 return NULL;
 }, &mutex);

 bthread_start_urgent(&tid2, NULL, [](void* arg) -> void* {
 bthread::Mutex* m = (bthread::Mutex*)arg;
 for (int i = 0; i < SWITCH_COUNT / 2; ++i) {
 std::unique_lock<bthread::Mutex> lock(*m);
 bthread_yield(); // 主动让出CPU
 }
 return NULL;
 }, &mutex);

 bthread_join(tid1, NULL);
 bthread_join(tid2, NULL);

 auto duration = std::chrono::duration_cast<std::chrono::microseconds>(
 std::chrono::high_resolution_clock::now() - start).count();

 LOG(INFO) << "Context switches: " << SWITCH_COUNT;
 LOG(INFO) << "Total time: " << duration << "us";
 LOG(INFO) << "Avg per switch: " << (duration / SWITCH_COUNT) << "us";
 }
};

3.1.10 在 RPC 服务中使用 bthread

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56


// 在 RPC 服务中使用 bthread 进行异步处理
class AsyncRpcService : public example::EchoService {
private:
 TaskProcessor task_processor_;

public:
 void Echo(google::protobuf::RpcController* cntl_base,
 const example::EchoRequest* request,
 example::EchoResponse* response,
 google::protobuf::Closure* done) {
 brpc::ClosureGuard done_guard(done);
 brpc::Controller* cntl = static_cast<brpc::Controller*>(cntl_base);

 // 当前已经在 bthread 中了
 LOG(INFO) << "Request received in bthread " << bthread_self();

 // 方法1：启动新的 bthread 处理耗时操作
 bthread_t worker_tid;
 struct TaskContext {
 std::string message;
 example::EchoResponse* response;
 google::protobuf::Closure* done;
 };

 TaskContext* ctx = new TaskContext{
 request->message(), response, done
 };

 if (bthread_start_background(&worker_tid, NULL,
 ProcessInBackground, ctx) == 0) {
 done_guard.release(); // 由 background bthread 负责调用 done
 }

 // 方法2：使用 execution_queue 异步处理
 // task_processor_.SubmitTask(request->message());
 }

private:
 static void* ProcessInBackground(void* arg) {
 std::unique_ptr<TaskContext> ctx((TaskContext*)arg);

 LOG(INFO) << "Processing in background bthread " << bthread_self();

 // 模拟耗时操作
 bthread_usleep(100000); // 100ms

 // 模拟访问数据库或其他服务
 std::string result = "Processed: " + ctx->message;
 ctx->response->set_message(result);

 // 完成 RPC
 ctx->done->Run();

 return NULL;
 }
};

3.2 Server（服务器）

Server 是 brpc 的服务端核心组件，特点：

自动协议识别：无需指定协议，自动识别客户端协议
多协议支持：一个端口可同时支持多种协议
线程模型：使用 bthread（协程）实现高并发

1
2
3
4
5
6


// 服务器配置示例
brpc::ServerOptions options;
options.num_threads = 16; // 工作线程数
options.max_concurrency = 0; // 最大并发请求数，0表示不限制
options.idle_timeout_sec = -1; // 连接空闲超时，-1表示不超时
options.internal_port = -1; // 内部监控端口

3.2 Channel（通道）

Channel 代表到服务器的通信线路，特点：

线程安全：可被所有线程共享
连接池：自动管理底层连接
负载均衡：支持多种负载均衡算法

1
2
3
4
5
6
7


// Channel 配置示例
brpc::ChannelOptions options;
options.protocol = brpc::PROTOCOL_BAIDU_STD; // 协议类型
options.connection_type = brpc::CONNECTION_TYPE_POOLED; // 连接池
options.timeout_ms = 100; // 超时时间
options.max_retry = 3; // 最大重试次数
options.backup_request_ms = -1; // backup request

3.3 Controller

Controller 用于控制单次 RPC 调用：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


brpc::Controller cntl;

// 设置请求参数
cntl.set_timeout_ms(200);
cntl.set_max_retry(2);
cntl.set_log_id(12345); // 用于追踪

// 发起调用
stub.Method(&cntl, &request, &response, NULL);

// 检查结果
if (cntl.Failed()) {
 LOG(ERROR) << "Error: " << cntl.ErrorText();
 // 获取详细错误信息
 int error_code = cntl.ErrorCode();
 // 获取响应延迟
 int64_t latency = cntl.latency_us();
}

四、进阶特性

4.1 异步调用

异步客户端

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


// 使用 Callback
void HandleResponse(brpc::Controller* cntl,
 example::EchoResponse* response) {
 if (!cntl->Failed()) {
 LOG(INFO) << "Got response: " << response->message();
 }
 delete cntl;
 delete response;
}

// 发起异步调用
brpc::Controller* cntl = new brpc::Controller();
example::EchoResponse* response = new example::EchoResponse();
google::protobuf::Closure* done = brpc::NewCallback(
 &HandleResponse, cntl, response);

stub.Echo(cntl, &request, response, done);

异步服务器

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


class AsyncEchoService : public example::EchoService {
public:
 void Echo(google::protobuf::RpcController* cntl_base,
 const example::EchoRequest* request,
 example::EchoResponse* response,
 google::protobuf::Closure* done) {
 brpc::ClosureGuard done_guard(done);

 // 启动异步任务
 bthread_t tid;
 bthread_start_background(&tid, NULL, ProcessRequest,
 new Context(request, response, done));

 // 释放 done_guard，让异步任务负责调用 done
 done_guard.release();
 }
};

4.2 流式 RPC

brpc 支持流式 RPC，适用于大数据传输：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


// 创建 Stream
brpc::StreamId stream_id;
brpc::StreamOptions stream_options;
if (brpc::StreamCreate(&stream_id, cntl, &stream_options) != 0) {
 LOG(ERROR) << "Fail to create stream";
 return;
}

// 写入数据
butil::IOBuf data;
data.append("streaming data");
brpc::StreamWrite(stream_id, data);

// 关闭 Stream
brpc::StreamClose(stream_id);

4.3 负载均衡

brpc 支持多种负载均衡算法：

1
2
3
4
5
6
7
8
9


// 设置负载均衡算法
options.load_balancer_name = "rr"; // round-robin
// options.load_balancer_name = "random"; // 随机
// options.load_balancer_name = "la"; // locality-aware
// options.load_balancer_name = "c_murmurhash"; // 一致性哈希

// 添加多个服务器节点
channel.Init("list://server1:8000,server2:8000,server3:8000",
 &options);

4.4 服务发现

集成服务发现系统：

1
2
3
4
5
6
7
8


// 使用 Naming Service
channel.Init("bns://service_name", &options);

// 或使用 Consul
channel.Init("consul://service_name", &options);

// 或使用文件
channel.Init("file://server_list.txt", &options);

五、性能优化

5.1 连接池优化

1
2
3
4
5
6
7
8


// 短连接（适合请求频率低的场景）
options.connection_type = brpc::CONNECTION_TYPE_SHORT;

// 连接池（推荐，适合大部分场景）
options.connection_type = brpc::CONNECTION_TYPE_POOLED;

// 单连接（适合流式或推送场景）
options.connection_type = brpc::CONNECTION_TYPE_SINGLE;

5.2 并发控制

1
2
3
4
5
6
7
8


// 服务器端并发控制
brpc::ServerOptions server_options;
server_options.max_concurrency = 1000; // 限制最大并发

// 客户端并发控制
brpc::ChannelOptions channel_options;
channel_options.max_retry = 3;
channel_options.backup_request_ms = 100; // 100ms 后发送 backup request

5.3 批量请求

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


// 使用 ParallelChannel 进行批量请求
brpc::ParallelChannel pchan;
pchan.Init(&options);

// 添加多个子 Channel
for (int i = 0; i < server_count; ++i) {
 brpc::Channel* sub_channel = new brpc::Channel;
 sub_channel->Init(server_addr[i], &options);
 pchan.AddChannel(sub_channel, brpc::OWNS_CHANNEL);
}

// 发送请求会自动分发到所有子 Channel

5.4 性能监控

1
2
3
4
5
6
7
8


// 启用内置服务
server.Start(8000, &options);

// 访问内置监控页面
// http://localhost:8000/status - 查看服务状态
// http://localhost:8000/vars - 查看统计变量
// http://localhost:8000/rpcz - 查看 RPC 详情
// http://localhost:8000/hotspots - 查看 CPU 热点

六、最佳实践

6.1 错误处理

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


// 完善的错误处理
void CallMethod() {
 brpc::Controller cntl;

 // 设置重试策略
 cntl.set_max_retry(3);

 // 调用
 stub.Method(&cntl, &request, &response, NULL);

 if (cntl.Failed()) {
 // 区分错误类型
 if (cntl.ErrorCode() == brpc::ENOSERVICE) {
 LOG(ERROR) << "Service not found";
 } else if (cntl.ErrorCode() == brpc::ENOMETHOD) {
 LOG(ERROR) << "Method not found";
 } else if (cntl.ErrorCode() == brpc::ETIMEOUT) {
 LOG(ERROR) << "Request timeout";
 } else {
 LOG(ERROR) << "Unknown error: " << cntl.ErrorText();
 }
 }
}

6.2 超时设置

1
2
3
4
5
6
7
8
9


// 分层超时设置
// 1. Channel 级别（默认超时）
channel_options.timeout_ms = 1000;

// 2. Controller 级别（单次请求超时）
cntl.set_timeout_ms(500);

// 3. Backup Request（用于长尾优化）
channel_options.backup_request_ms = 200;

6.3 日志和追踪

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


// 设置 log_id 用于全链路追踪
cntl.set_log_id(GenerateLogId());

// 自定义附件
cntl.request_attachment().append("custom_data");

// 获取详细时间统计
LOG(INFO) << "Latency: " << cntl.latency_us() << "us"
 << " Send: " << cntl.send_latency_us() << "us"
 << " Receive: " << cntl.receive_latency_us() << "us";

6.4 安全通信

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


// 启用 SSL/TLS
brpc::ChannelOptions options;
options.mutable_ssl_options()->client_cert = "client.crt";
options.mutable_ssl_options()->client_key = "client.key";
options.mutable_ssl_options()->ca_cert = "ca.crt";

// 服务器端 SSL
brpc::ServerOptions server_options;
server_options.mutable_ssl_options()->cert = "server.crt";
server_options.mutable_ssl_options()->key = "server.key";

七、实战案例

7.1 实现限流中间件

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


class RateLimiter : public brpc::Interceptor {
private:
 std::atomic<int> qps_count{0};
 int max_qps;

public:
 RateLimiter(int max_qps) : max_qps(max_qps) {}

 void BeforeRpc(brpc::Controller* cntl) override {
 if (qps_count.fetch_add(1) > max_qps) {
 cntl->SetFailed(brpc::ELIMIT, "Rate limit exceeded");
 cntl->SendResponse();
 }
 }

 void AfterRpc(brpc::Controller* cntl) override {
 qps_count.fetch_sub(1);
 }
};

// 使用
server.AddInterceptor(new RateLimiter(10000));

7.2 实现熔断器

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


class CircuitBreaker {
private:
 std::atomic<int> failure_count{0};
 std::atomic<bool> is_open{false};
 int threshold = 10;

public:
 bool ShouldPass() {
 if (is_open.load()) {
 return false; // 熔断状态，拒绝请求
 }
 return true;
 }

 void OnSuccess() {
 failure_count.store(0);
 is_open.store(false);
 }

 void OnFailure() {
 if (failure_count.fetch_add(1) >= threshold) {
 is_open.store(true); // 触发熔断
 // 启动定时器，一段时间后尝试恢复
 }
 }
};

7.3 实现请求缓存

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


class CachedService : public example::EchoService {
private:
 std::unordered_map<std::string, std::string> cache;
 std::mutex cache_mutex;

public:
 void Echo(google::protobuf::RpcController* cntl_base,
 const example::EchoRequest* request,
 example::EchoResponse* response,
 google::protobuf::Closure* done) {
 brpc::ClosureGuard done_guard(done);

 // 查询缓存
 {
 std::lock_guard<std::mutex> lock(cache_mutex);
 auto it = cache.find(request->message());
 if (it != cache.end()) {
 response->set_message(it->second);
 return;
 }
 }

 // 处理请求
 std::string result = ProcessRequest(request->message());
 response->set_message(result);

 // 更新缓存
 {
 std::lock_guard<std::mutex> lock(cache_mutex);
 cache[request->message()] = result;
 }
 }
};

八、故障排查

8.1 常见问题

连接问题

1
2
3
4
5
6
7


// 检查连接状态
if (channel.CheckHealth() != 0) {
 LOG(ERROR) << "Channel is unhealthy";
}

// 获取连接统计
LOG(INFO) << "Connection count: " << channel.ConnectionCount();

性能问题

1
2
3
4
5
6
7
8
9


// 使用内置性能分析工具
// 1. CPU profiling
// 访问 http://server:port/hotspots

// 2. 内存分析
// 访问 http://server:port/heap

// 3. RPC 追踪
// 访问 http://server:port/rpcz

8.2 调试技巧

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


// 开启详细日志
FLAGS_v = 4; // 设置日志级别

// 打印请求响应
class DebugInterceptor : public brpc::Interceptor {
 void BeforeRpc(brpc::Controller* cntl) override {
 LOG(INFO) << "Request: " << cntl->request_attachment();
 }

 void AfterRpc(brpc::Controller* cntl) override {
 LOG(INFO) << "Response: " << cntl->response_attachment();
 }
};

九、与其他框架对比

9.1 brpc vs gRPC

特性	brpc	gRPC
性能	极高（QPS 高 10000+）	高
协议支持	丰富（HTTP/H2/Redis/MC等）	主要是 HTTP/2
负载均衡	内置多种算法	需要额外组件
服务发现	内置支持	需要额外组件
调试工具	丰富的内置工具	相对较少
学习曲线	适中	较陡
社区	Apache 社区	Google 主导

9.2 选择建议

选择 brpc：
- 需要极致性能（高并发、低延迟）
- 需要支持多种协议
- 在百度云或类似环境
- 需要丰富的调试工具
选择 gRPC：
- 跨语言支持更重要
- 与 Google 生态集成
- 团队已有 gRPC 经验

十、学习资源

10.1 官方资源

10.2 示例代码

example/echo_c++/ - Echo 服务示例
example/multi_threaded_echo_c++/ - 多线程示例
example/streaming_echo_c++/ - 流式 RPC 示例
example/parallel_echo_c++/ - 并行请求示例

10.3 进阶学习路径

基础阶段（1-2周）
- 理解 RPC 基本概念
- 完成 Echo 示例
- 掌握基本的 Server/Channel 使用
进阶阶段（2-4周）
- 学习异步编程模型
- 掌握流式 RPC
- 理解负载均衡和服务发现
高级阶段（1-2月）
- 源码阅读（bthread 实现）
- 性能调优实践
- 自定义协议开发
专家阶段
- 参与开源贡献
- 大规模分布式系统实践
- 框架二次开发

十一、总结

brpc 作为百度开源的工业级 RPC 框架，在性能、易用性和功能完整性方面都达到了业界领先水平。其优势不仅体现在极致的性能优化上，更在于其完善的生态和丰富的功能特性。

对于需要构建高性能分布式系统的团队，brpc 是一个值得认真考虑的选择。通过本指南的学习，相信你已经对 brpc 有了全面的认识，能够在实际项目中灵活运用。

记住，掌握一个框架的关键在于：

理解原理：知其然更要知其所以然
动手实践：从简单示例开始，逐步深入
阅读源码：深入理解实现细节
解决问题：在实践中积累经验

祝你在 brpc 的学习之路上越走越远！

]]>

数据结构入门教程：散列表数据结构详解与Java实现

lesshash@blog.lesshash.com/ (lesshash) — Tue, 28 Jan 2025 16:00:00 +0800

🗂️ 引言：快速存取的魔法盒

想象一下图书馆的分类系统：每本书都有一个独特的编号，通过这个编号就能快速找到书架上的确切位置。**散列表（Hash Table）**就是数据世界的图书馆——通过巧妙的编号系统（哈希函数），让我们能够在 O(1) 时间内存取数据！

散列表是一种基于键值对存储的数据结构，它使用哈希函数将键映射到数组的特定位置。这种"以空间换时间"的设计让散列表成为现代编程中最重要的数据结构之一。

流程图表

关系流向：

1
2
3
4
5


A["键值对 (Key, Value)"] → B["哈希函数 hash()"]
B → C["数组索引"]
C → D["存储位置"]
E["index 0"] → F["(apple, 🍎)"]
G["index 1"] → H["空"]

🔑 哈希函数设计

哈希函数是散列表的核心，决定了性能和冲突率。

常见哈希函数

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120


/**
 * 哈希函数集合
 * 演示不同类型的哈希函数设计
 */
public class HashFunctions {

 /**
 * 除法散列法
 * 最简单的哈希函数：h(k) = k mod m
 */
 public static int divisionHash(int key, int tableSize) {
 return Math.abs(key) % tableSize;
 }

 /**
 * 乘法散列法
 * h(k) = floor(m * ((k * A) mod 1))
 * 其中 A = (√5 - 1) / 2 ≈ 0.618 (黄金比例的小数部分)
 */
 public static int multiplicationHash(int key, int tableSize) {
 double A = (Math.sqrt(5) - 1) / 2; // 黄金比例
 double temp = key * A;
 double fractionalPart = temp - Math.floor(temp);
 return (int) Math.floor(tableSize * fractionalPart);
 }

 /**
 * 字符串哈希函数 - DJB2算法
 * 经典的字符串哈希算法，冲突率低
 */
 public static int djb2Hash(String str, int tableSize) {
 long hash = 5381;
 for (char c : str.toCharArray()) {
 hash = ((hash << 5) + hash) + c; // hash * 33 + c
 }
 return (int) (Math.abs(hash) % tableSize);
 }

 /**
 * 字符串哈希函数 - SDBM算法
 * 另一种高质量的字符串哈希算法
 */
 public static int sdbmHash(String str, int tableSize) {
 long hash = 0;
 for (char c : str.toCharArray()) {
 hash = c + (hash << 6) + (hash << 16) - hash;
 }
 return (int) (Math.abs(hash) % tableSize);
 }

 /**
 * FNV-1a哈希算法
 * 快速且分布均匀的哈希算法
 */
 public static int fnv1aHash(String str, int tableSize) {
 final int FNV_PRIME = 16777619;
 final int FNV_OFFSET_BASIS = (int) 2166136261L;

 int hash = FNV_OFFSET_BASIS;
 for (char c : str.toCharArray()) {
 hash ^= c;
 hash *= FNV_PRIME;
 }
 return Math.abs(hash) % tableSize;
 }

 /**
 * 通用哈希函数
 * 使用随机参数减少最坏情况的出现
 */
 public static class UniversalHashFunction {
 private final int a, b, p, m;

 public UniversalHashFunction(int tableSize) {
 Random random = new Random();
 this.p = 1000003; // 一个大质数
 this.m = tableSize;
 this.a = random.nextInt(p - 1) + 1; // 1 <= a <= p-1
 this.b = random.nextInt(p); // 0 <= b <= p-1
 }

 public int hash(int key) {
 return ((a * key + b) % p) % m;
 }
 }

 /**
 * 测试哈希函数的分布均匀性
 */
 public static void testHashDistribution() {
 String[] testStrings = {
 "apple", "banana", "cherry", "date", "elderberry",
 "fig", "grape", "honeydew", "kiwi", "lemon",
 "mango", "nectarine", "orange", "papaya", "quince"
 };

 int tableSize = 11;
 int[] distribution = new int[tableSize];

 System.out.println("哈希函数分布测试:");
 System.out.println("表大小: " + tableSize);
 System.out.println("\n字符串\t\tDJB2\tSDBM\tFNV1a");
 System.out.println("-".repeat(50));

 for (String str : testStrings) {
 int djb2 = djb2Hash(str, tableSize);
 int sdbm = sdbmHash(str, tableSize);
 int fnv1a = fnv1aHash(str, tableSize);

 distribution[djb2]++;

 System.out.printf("%-12s\t%d\t%d\t%d%n", str, djb2, sdbm, fnv1a);
 }

 System.out.println("\nDJB2算法分布统计:");
 for (int i = 0; i < tableSize; i++) {
 System.out.println("位置 " + i + ": " + distribution[i] + " 个元素");
 }
 }
}

🛠️ 冲突解决策略

当不同的键映射到同一个数组位置时，就发生了冲突。

1. 链地址法（Separate Chaining）

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206


/**
 * 使用链地址法解决冲突的散列表
 * 每个数组位置维护一个链表
 */
public class ChainedHashTable<K, V> {

 /**
 * 键值对节点
 */
 private static class Entry<K, V> {
 K key;
 V value;
 Entry<K, V> next;

 Entry(K key, V value) {
 this.key = key;
 this.value = value;
 }

 @Override
 public String toString() {
 return key + "=" + value;
 }
 }

 private Entry<K, V>[] table; // 散列表数组
 private int size; // 元素个数
 private int capacity; // 表容量
 private static final double LOAD_FACTOR_THRESHOLD = 0.75; // 负载因子阈值

 @SuppressWarnings("unchecked")
 public ChainedHashTable(int initialCapacity) {
 this.capacity = initialCapacity;
 this.table = new Entry[capacity];
 this.size = 0;
 }

 public ChainedHashTable() {
 this(16);
 }

 /**
 * 哈希函数
 */
 private int hash(K key) {
 if (key == null) return 0;
 return Math.abs(key.hashCode()) % capacity;
 }

 /**
 * 插入或更新键值对
 */
 public V put(K key, V value) {
 int index = hash(key);
 Entry<K, V> entry = table[index];

 // 在链表中查找键
 while (entry != null) {
 if (key.equals(entry.key)) {
 V oldValue = entry.value;
 entry.value = value;
 System.out.println("更新: " + key + " -> " + value + " (位置: " + index + ")");
 return oldValue;
 }
 entry = entry.next;
 }

 // 键不存在，插入新节点到链表头部
 Entry<K, V> newEntry = new Entry<>(key, value);
 newEntry.next = table[index];
 table[index] = newEntry;
 size++;

 System.out.println("插入: " + key + " -> " + value + " (位置: " + index + ")");

 // 检查是否需要扩容
 if ((double) size / capacity > LOAD_FACTOR_THRESHOLD) {
 resize();
 }

 return null;
 }

 /**
 * 获取值
 */
 public V get(K key) {
 int index = hash(key);
 Entry<K, V> entry = table[index];

 while (entry != null) {
 if (key.equals(entry.key)) {
 System.out.println("找到: " + key + " -> " + entry.value + " (位置: " + index + ")");
 return entry.value;
 }
 entry = entry.next;
 }

 System.out.println("未找到: " + key);
 return null;
 }

 /**
 * 删除键值对
 */
 public V remove(K key) {
 int index = hash(key);
 Entry<K, V> entry = table[index];
 Entry<K, V> prev = null;

 while (entry != null) {
 if (key.equals(entry.key)) {
 if (prev == null) {
 table[index] = entry.next;
 } else {
 prev.next = entry.next;
 }
 size--;
 System.out.println("删除: " + key + " -> " + entry.value + " (位置: " + index + ")");
 return entry.value;
 }
 prev = entry;
 entry = entry.next;
 }

 System.out.println("删除失败，未找到: " + key);
 return null;
 }

 /**
 * 扩容操作
 */
 @SuppressWarnings("unchecked")
 private void resize() {
 System.out.println("开始扩容: " + capacity + " -> " + (capacity * 2));

 Entry<K, V>[] oldTable = table;
 int oldCapacity = capacity;

 capacity *= 2;
 table = new Entry[capacity];
 size = 0;

 // 重新插入所有元素
 for (int i = 0; i < oldCapacity; i++) {
 Entry<K, V> entry = oldTable[i];
 while (entry != null) {
 put(entry.key, entry.value);
 entry = entry.next;
 }
 }

 System.out.println("扩容完成，新容量: " + capacity);
 }

 /**
 * 获取负载因子
 */
 public double getLoadFactor() {
 return (double) size / capacity;
 }

 /**
 * 显示散列表状态
 */
 public void display() {
 System.out.println("\n散列表状态 (容量: " + capacity + ", 大小: " + size +
 ", 负载因子: " + String.format("%.2f", getLoadFactor()) + "):");

 for (int i = 0; i < capacity; i++) {
 System.out.print("位置 " + i + ": ");
 Entry<K, V> entry = table[i];

 if (entry == null) {
 System.out.println("空");
 } else {
 while (entry != null) {
 System.out.print(entry);
 if (entry.next != null) System.out.print(" -> ");
 entry = entry.next;
 }
 System.out.println();
 }
 }
 }

 /**
 * 获取最长链的长度（用于性能分析）
 */
 public int getMaxChainLength() {
 int maxLength = 0;
 for (int i = 0; i < capacity; i++) {
 int length = 0;
 Entry<K, V> entry = table[i];
 while (entry != null) {
 length++;
 entry = entry.next;
 }
 maxLength = Math.max(maxLength, length);
 }
 return maxLength;
 }

 public int size() { return size; }
 public boolean isEmpty() { return size == 0; }
}

2. 开放寻址法（Open Addressing）

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248


/**
 * 使用开放寻址法解决冲突的散列表
 * 支持线性探测、二次探测和双重散列
 */
public class OpenAddressingHashTable<K, V> {

 /**
 * 探测方法枚举
 */
 public enum ProbeType {
 LINEAR, // 线性探测
 QUADRATIC, // 二次探测
 DOUBLE_HASH // 双重散列
 }

 /**
 * 键值对条目
 */
 private static class Entry<K, V> {
 K key;
 V value;
 boolean isDeleted; // 懒删除标记

 Entry(K key, V value) {
 this.key = key;
 this.value = value;
 this.isDeleted = false;
 }

 @Override
 public String toString() {
 return isDeleted ? "[已删除]" : key + "=" + value;
 }
 }

 private Entry<K, V>[] table;
 private int size;
 private int capacity;
 private ProbeType probeType;
 private static final double LOAD_FACTOR_THRESHOLD = 0.5;

 @SuppressWarnings("unchecked")
 public OpenAddressingHashTable(int initialCapacity, ProbeType probeType) {
 this.capacity = initialCapacity;
 this.table = new Entry[capacity];
 this.size = 0;
 this.probeType = probeType;
 }

 public OpenAddressingHashTable(ProbeType probeType) {
 this(16, probeType);
 }

 /**
 * 主哈希函数
 */
 private int hash1(K key) {
 if (key == null) return 0;
 return Math.abs(key.hashCode()) % capacity;
 }

 /**
 * 辅助哈希函数（用于双重散列）
 */
 private int hash2(K key) {
 if (key == null) return 1;
 return 7 - (Math.abs(key.hashCode()) % 7);
 }

 /**
 * 探测函数
 */
 private int probe(K key, int attempt) {
 int h1 = hash1(key);

 switch (probeType) {
 case LINEAR:
 return (h1 + attempt) % capacity;

 case QUADRATIC:
 return (h1 + attempt * attempt) % capacity;

 case DOUBLE_HASH:
 int h2 = hash2(key);
 return (h1 + attempt * h2) % capacity;

 default:
 return h1;
 }
 }

 /**
 * 插入或更新键值对
 */
 public V put(K key, V value) {
 if ((double) size / capacity >= LOAD_FACTOR_THRESHOLD) {
 resize();
 }

 for (int attempt = 0; attempt < capacity; attempt++) {
 int index = probe(key, attempt);
 Entry<K, V> entry = table[index];

 // 位置为空或已删除，可以插入
 if (entry == null || entry.isDeleted) {
 table[index] = new Entry<>(key, value);
 if (entry == null || entry.isDeleted) {
 size++;
 }
 System.out.println("插入: " + key + " -> " + value +
 " (位置: " + index + ", 探测次数: " + (attempt + 1) + ")");
 return null;
 }

 // 键已存在，更新值
 if (key.equals(entry.key)) {
 V oldValue = entry.value;
 entry.value = value;
 System.out.println("更新: " + key + " -> " + value +
 " (位置: " + index + ", 探测次数: " + (attempt + 1) + ")");
 return oldValue;
 }
 }

 throw new IllegalStateException("散列表已满，无法插入");
 }

 /**
 * 获取值
 */
 public V get(K key) {
 for (int attempt = 0; attempt < capacity; attempt++) {
 int index = probe(key, attempt);
 Entry<K, V> entry = table[index];

 if (entry == null) {
 System.out.println("未找到: " + key + " (探测次数: " + (attempt + 1) + ")");
 return null;
 }

 if (!entry.isDeleted && key.equals(entry.key)) {
 System.out.println("找到: " + key + " -> " + entry.value +
 " (位置: " + index + ", 探测次数: " + (attempt + 1) + ")");
 return entry.value;
 }
 }

 System.out.println("未找到: " + key + " (已探测整个表)");
 return null;
 }

 /**
 * 删除键值对（懒删除）
 */
 public V remove(K key) {
 for (int attempt = 0; attempt < capacity; attempt++) {
 int index = probe(key, attempt);
 Entry<K, V> entry = table[index];

 if (entry == null) {
 System.out.println("删除失败，未找到: " + key);
 return null;
 }

 if (!entry.isDeleted && key.equals(entry.key)) {
 entry.isDeleted = true;
 size--;
 System.out.println("删除: " + key + " -> " + entry.value +
 " (位置: " + index + ", 探测次数: " + (attempt + 1) + ")");
 return entry.value;
 }
 }

 System.out.println("删除失败，未找到: " + key);
 return null;
 }

 /**
 * 扩容并重新散列
 */
 @SuppressWarnings("unchecked")
 private void resize() {
 System.out.println("开始扩容: " + capacity + " -> " + (capacity * 2));

 Entry<K, V>[] oldTable = table;
 int oldCapacity = capacity;

 capacity *= 2;
 table = new Entry[capacity];
 size = 0;

 // 重新插入所有有效元素
 for (int i = 0; i < oldCapacity; i++) {
 Entry<K, V> entry = oldTable[i];
 if (entry != null && !entry.isDeleted) {
 put(entry.key, entry.value);
 }
 }

 System.out.println("扩容完成，新容量: " + capacity);
 }

 /**
 * 显示散列表状态
 */
 public void display() {
 System.out.println("\n散列表状态 (" + probeType + " 探测, 容量: " + capacity +
 ", 大小: " + size + ", 负载因子: " +
 String.format("%.2f", (double) size / capacity) + "):");

 for (int i = 0; i < capacity; i++) {
 Entry<K, V> entry = table[i];
 if (entry == null) {
 System.out.println("位置 " + i + ": 空");
 } else {
 System.out.println("位置 " + i + ": " + entry);
 }
 }
 }

 /**
 * 计算平均探测次数
 */
 public double getAverageProbeCount() {
 int totalProbes = 0;
 int validEntries = 0;

 for (int i = 0; i < capacity; i++) {
 Entry<K, V> entry = table[i];
 if (entry != null && !entry.isDeleted) {
 // 重新计算这个元素的探测次数
 for (int attempt = 0; attempt < capacity; attempt++) {
 int index = probe(entry.key, attempt);
 if (index == i) {
 totalProbes += attempt + 1;
 validEntries++;
 break;
 }
 }
 }
 }

 return validEntries == 0 ? 0 : (double) totalProbes / validEntries;
 }

 public int size() { return size; }
 public boolean isEmpty() { return size == 0; }
}

🌟 高级散列表实现

Robin Hood 散列

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177


/**
 * Robin Hood 散列实现
 * 通过"劫富济贫"的策略优化探测距离
 */
public class RobinHoodHashTable<K, V> {

 private static class Entry<K, V> {
 K key;
 V value;
 int distance; // 探测距离

 Entry(K key, V value, int distance) {
 this.key = key;
 this.value = value;
 this.distance = distance;
 }

 @Override
 public String toString() {
 return key + "=" + value + "(d:" + distance + ")";
 }
 }

 private Entry<K, V>[] table;
 private int size;
 private int capacity;
 private static final double LOAD_FACTOR_THRESHOLD = 0.75;

 @SuppressWarnings("unchecked")
 public RobinHoodHashTable(int initialCapacity) {
 this.capacity = initialCapacity;
 this.table = new Entry[capacity];
 this.size = 0;
 }

 private int hash(K key) {
 if (key == null) return 0;
 return Math.abs(key.hashCode()) % capacity;
 }

 /**
 * 插入键值对
 */
 public V put(K key, V value) {
 if ((double) size / capacity >= LOAD_FACTOR_THRESHOLD) {
 resize();
 }

 int hash = hash(key);
 Entry<K, V> newEntry = new Entry<>(key, value, 0);

 return robinHoodInsert(newEntry, hash);
 }

 /**
 * Robin Hood 插入算法
 */
 private V robinHoodInsert(Entry<K, V> entry, int startPos) {
 int pos = startPos;

 while (true) {
 int actualPos = pos % capacity;

 // 位置为空，直接插入
 if (table[actualPos] == null) {
 entry.distance = pos - startPos;
 table[actualPos] = entry;
 size++;
 System.out.println("插入: " + entry.key + " -> " + entry.value +
 " (位置: " + actualPos + ", 距离: " + entry.distance + ")");
 return null;
 }

 // 键已存在，更新值
 if (entry.key.equals(table[actualPos].key)) {
 V oldValue = table[actualPos].value;
 table[actualPos].value = entry.value;
 return oldValue;
 }

 // Robin Hood 策略：如果新元素的距离大于当前元素的距离，交换它们
 entry.distance = pos - startPos;
 if (entry.distance > table[actualPos].distance) {
 Entry<K, V> displaced = table[actualPos];
 table[actualPos] = entry;

 // 继续为被替换的元素寻找位置
 entry = displaced;
 startPos = actualPos - displaced.distance;

 System.out.println("Robin Hood 交换: " + entry.key + " 被 " +
 table[actualPos].key + " 替换 (位置: " + actualPos + ")");
 }

 pos++;
 }
 }

 /**
 * 获取值
 */
 public V get(K key) {
 int hash = hash(key);
 int pos = hash;

 while (true) {
 int actualPos = pos % capacity;
 Entry<K, V> entry = table[actualPos];

 if (entry == null) {
 return null;
 }

 if (entry.key.equals(key)) {
 return entry.value;
 }

 // 如果当前探测距离超过了元素的距离，说明元素不存在
 int currentDistance = pos - hash;
 if (currentDistance > entry.distance) {
 return null;
 }

 pos++;
 }
 }

 /**
 * 扩容
 */
 @SuppressWarnings("unchecked")
 private void resize() {
 Entry<K, V>[] oldTable = table;
 int oldCapacity = capacity;

 capacity *= 2;
 table = new Entry[capacity];
 size = 0;

 for (int i = 0; i < oldCapacity; i++) {
 Entry<K, V> entry = oldTable[i];
 if (entry != null) {
 put(entry.key, entry.value);
 }
 }
 }

 /**
 * 显示散列表状态
 */
 public void display() {
 System.out.println("\nRobin Hood 散列表状态:");
 for (int i = 0; i < capacity; i++) {
 Entry<K, V> entry = table[i];
 if (entry == null) {
 System.out.println("位置 " + i + ": 空");
 } else {
 System.out.println("位置 " + i + ": " + entry);
 }
 }

 // 计算平均探测距离
 double avgDistance = 0;
 int count = 0;
 for (int i = 0; i < capacity; i++) {
 Entry<K, V> entry = table[i];
 if (entry != null) {
 avgDistance += entry.distance;
 count++;
 }
 }
 if (count > 0) {
 avgDistance /= count;
 System.out.println("平均探测距离: " + String.format("%.2f", avgDistance));
 }
 }
}

一致性哈希

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211


/**
 * 一致性哈希实现
 * 用于分布式系统中的负载均衡
 */
public class ConsistentHashing {

 /**
 * 虚拟节点类
 */
 private static class VirtualNode {
 String nodeId;
 int hash;

 VirtualNode(String nodeId, int hash) {
 this.nodeId = nodeId;
 this.hash = hash;
 }

 @Override
 public String toString() {
 return nodeId + "@" + hash;
 }
 }

 private final TreeMap<Integer, String> ring; // 哈希环
 private final Set<String> nodes; // 物理节点
 private final int virtualNodesPerNode; // 每个物理节点的虚拟节点数

 public ConsistentHashing(int virtualNodesPerNode) {
 this.ring = new TreeMap<>();
 this.nodes = new HashSet<>();
 this.virtualNodesPerNode = virtualNodesPerNode;
 }

 /**
 * 添加节点
 */
 public void addNode(String nodeId) {
 nodes.add(nodeId);

 // 为每个物理节点创建多个虚拟节点
 for (int i = 0; i < virtualNodesPerNode; i++) {
 String virtualNodeId = nodeId + "#" + i;
 int hash = hash(virtualNodeId);
 ring.put(hash, nodeId);
 System.out.println("添加虚拟节点: " + virtualNodeId + " -> " + hash);
 }

 System.out.println("节点 " + nodeId + " 添加完成，创建了 " +
 virtualNodesPerNode + " 个虚拟节点");
 }

 /**
 * 移除节点
 */
 public void removeNode(String nodeId) {
 if (!nodes.contains(nodeId)) {
 return;
 }

 nodes.remove(nodeId);

 // 移除所有相关的虚拟节点
 Iterator<Map.Entry<Integer, String>> iterator = ring.entrySet().iterator();
 int removedCount = 0;

 while (iterator.hasNext()) {
 Map.Entry<Integer, String> entry = iterator.next();
 if (entry.getValue().equals(nodeId)) {
 iterator.remove();
 removedCount++;
 }
 }

 System.out.println("节点 " + nodeId + " 移除完成，删除了 " + removedCount + " 个虚拟节点");
 }

 /**
 * 获取负责处理指定键的节点
 */
 public String getNode(String key) {
 if (ring.isEmpty()) {
 return null;
 }

 int hash = hash(key);
 System.out.println("查找键: " + key + " (hash: " + hash + ")");

 // 找到第一个大于等于该hash值的节点
 Map.Entry<Integer, String> entry = ring.ceilingEntry(hash);

 if (entry == null) {
 // 如果没找到，返回环上的第一个节点（环形特性）
 entry = ring.firstEntry();
 }

 System.out.println(" -> 路由到节点: " + entry.getValue());
 return entry.getValue();
 }

 /**
 * 简单哈希函数
 */
 private int hash(String input) {
 return Math.abs(input.hashCode());
 }

 /**
 * 分析数据分布
 */
 public void analyzeDistribution(String[] keys) {
 Map<String, Integer> distribution = new HashMap<>();

 // 初始化计数器
 for (String node : nodes) {
 distribution.put(node, 0);
 }

 // 统计每个键的分布
 for (String key : keys) {
 String node = getNode(key);
 if (node != null) {
 distribution.put(node, distribution.get(node) + 1);
 }
 }

 // 显示分布结果
 System.out.println("\n数据分布分析:");
 System.out.println("总键数: " + keys.length);

 for (Map.Entry<String, Integer> entry : distribution.entrySet()) {
 double percentage = (double) entry.getValue() / keys.length * 100;
 System.out.printf("节点 %s: %d 个键 (%.1f%%)%n",
 entry.getKey(), entry.getValue(), percentage);
 }

 // 计算标准差
 double mean = (double) keys.length / nodes.size();
 double variance = 0;

 for (int count : distribution.values()) {
 variance += Math.pow(count - mean, 2);
 }
 variance /= nodes.size();
 double stdDev = Math.sqrt(variance);

 System.out.printf("均值: %.1f, 标准差: %.1f%n", mean, stdDev);
 }

 /**
 * 显示哈希环状态
 */
 public void displayRing() {
 System.out.println("\n哈希环状态:");
 System.out.println("物理节点: " + nodes);
 System.out.println("环上虚拟节点数: " + ring.size());

 // 显示环上的虚拟节点分布
 for (Map.Entry<Integer, String> entry : ring.entrySet()) {
 System.out.println(" " + entry.getKey() + " -> " + entry.getValue());
 }
 }

 /**
 * 模拟节点故障时的数据迁移
 */
 public void simulateNodeFailure(String failedNode, String[] keys) {
 System.out.println("\n模拟节点故障: " + failedNode);

 // 记录故障前的分布
 Map<String, Set<String>> beforeFailure = new HashMap<>();
 for (String node : nodes) {
 beforeFailure.put(node, new HashSet<>());
 }

 for (String key : keys) {
 String node = getNode(key);
 if (node != null) {
 beforeFailure.get(node).add(key);
 }
 }

 // 移除故障节点
 Set<String> affectedKeys = beforeFailure.get(failedNode);
 removeNode(failedNode);

 // 计算迁移后的分布
 Map<String, Set<String>> afterFailure = new HashMap<>();
 for (String node : nodes) {
 afterFailure.put(node, new HashSet<>());
 }

 for (String key : affectedKeys) {
 String newNode = getNode(key);
 if (newNode != null) {
 afterFailure.get(newNode).add(key);
 }
 }

 // 分析迁移结果
 System.out.println("受影响的键数: " + affectedKeys.size());
 System.out.println("键的重新分布:");

 for (Map.Entry<String, Set<String>> entry : afterFailure.entrySet()) {
 if (!entry.getValue().isEmpty()) {
 System.out.println(" 节点 " + entry.getKey() + ": +" +
 entry.getValue().size() + " 个键");
 }
 }
 }
}

🧪 完整测试示例

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190


/**
 * 散列表数据结构综合测试
 */
public class HashTableTest {
 public static void main(String[] args) {
 System.out.println("=== 散列表数据结构综合测试 ===");

 testHashFunctions();
 testChainedHashTable();
 testOpenAddressing();
 testRobinHoodHashing();
 testConsistentHashing();
 performanceComparison();
 }

 private static void testHashFunctions() {
 System.out.println("\n1. 哈希函数测试:");
 HashFunctions.testHashDistribution();

 // 测试通用哈希函数
 HashFunctions.UniversalHashFunction uhf = new HashFunctions.UniversalHashFunction(11);
 System.out.println("\n通用哈希函数示例:");
 for (int i = 1; i <= 10; i++) {
 System.out.println("hash(" + i + ") = " + uhf.hash(i));
 }
 }

 private static void testChainedHashTable() {
 System.out.println("\n2. 链地址法散列表测试:");

 ChainedHashTable<String, Integer> table = new ChainedHashTable<>(7);

 // 插入数据
 String[] keys = {"apple", "banana", "cherry", "date", "elderberry", "fig", "grape"};
 for (int i = 0; i < keys.length; i++) {
 table.put(keys[i], i + 1);
 }

 table.display();
 System.out.println("最长链长度: " + table.getMaxChainLength());

 // 测试查找和删除
 System.out.println("\n查找测试:");
 table.get("apple");
 table.get("banana");
 table.get("notfound");

 System.out.println("\n删除测试:");
 table.remove("cherry");
 table.display();
 }

 private static void testOpenAddressing() {
 System.out.println("\n3. 开放寻址法测试:");

 // 测试不同探测方法
 OpenAddressingHashTable.ProbeType[] probeTypes = {
 OpenAddressingHashTable.ProbeType.LINEAR,
 OpenAddressingHashTable.ProbeType.QUADRATIC,
 OpenAddressingHashTable.ProbeType.DOUBLE_HASH
 };

 for (OpenAddressingHashTable.ProbeType probeType : probeTypes) {
 System.out.println("\n" + probeType + " 探测:");
 OpenAddressingHashTable<Integer, String> table =
 new OpenAddressingHashTable<>(7, probeType);

 for (int i = 1; i <= 5; i++) {
 table.put(i * 7, "Value" + i); // 故意造成冲突
 }

 table.display();
 System.out.println("平均探测次数: " +
 String.format("%.2f", table.getAverageProbeCount()));
 }
 }

 private static void testRobinHoodHashing() {
 System.out.println("\n4. Robin Hood 散列测试:");

 RobinHoodHashTable<Integer, String> table = new RobinHoodHashTable<>(7);

 for (int i = 1; i <= 5; i++) {
 table.put(i * 7, "Value" + i); // 故意造成冲突
 }

 table.display();
 }

 private static void testConsistentHashing() {
 System.out.println("\n5. 一致性哈希测试:");

 ConsistentHashing ch = new ConsistentHashing(3); // 每个节点3个虚拟节点

 // 添加节点
 ch.addNode("Server1");
 ch.addNode("Server2");
 ch.addNode("Server3");

 // 生成测试数据
 String[] keys = new String[30];
 for (int i = 0; i < keys.length; i++) {
 keys[i] = "key" + i;
 }

 // 分析初始分布
 ch.analyzeDistribution(keys);

 // 模拟节点故障
 ch.simulateNodeFailure("Server2", keys);
 }

 private static void performanceComparison() {
 System.out.println("\n6. 性能对比测试:");

 int[] sizes = {1000, 10000, 100000};

 System.out.println("数据规模\t链地址法\t开放寻址\tJava HashMap");
 System.out.println("-".repeat(60));

 for (int size : sizes) {
 // 生成测试数据
 String[] keys = new String[size];
 for (int i = 0; i < size; i++) {
 keys[i] = "key" + i;
 }

 // 测试链地址法
 long time1 = testChainedPerformance(keys);

 // 测试开放寻址法
 long time2 = testOpenAddressingPerformance(keys);

 // 测试Java HashMap
 long time3 = testJavaHashMapPerformance(keys);

 System.out.printf("%d\t\t%.2fms\t\t%.2fms\t\t%.2fms%n",
 size,
 time1 / 1_000_000.0,
 time2 / 1_000_000.0,
 time3 / 1_000_000.0);
 }
 }

 private static long testChainedPerformance(String[] keys) {
 ChainedHashTable<String, Integer> table = new ChainedHashTable<>();
 long start = System.nanoTime();

 for (int i = 0; i < keys.length; i++) {
 table.put(keys[i], i);
 }

 for (String key : keys) {
 table.get(key);
 }

 return System.nanoTime() - start;
 }

 private static long testOpenAddressingPerformance(String[] keys) {
 OpenAddressingHashTable<String, Integer> table =
 new OpenAddressingHashTable<>(OpenAddressingHashTable.ProbeType.LINEAR);
 long start = System.nanoTime();

 for (int i = 0; i < keys.length; i++) {
 table.put(keys[i], i);
 }

 for (String key : keys) {
 table.get(key);
 }

 return System.nanoTime() - start;
 }

 private static long testJavaHashMapPerformance(String[] keys) {
 HashMap<String, Integer> map = new HashMap<>();
 long start = System.nanoTime();

 for (int i = 0; i < keys.length; i++) {
 map.put(keys[i], i);
 }

 for (String key : keys) {
 map.get(key);
 }

 return System.nanoTime() - start;
 }
}

🎯 总结

散列表是现代计算机科学中最重要的数据结构之一，它的优雅在于简单而高效：

核心优势

平均 O(1) 性能：查找、插入、删除操作的平均时间复杂度
实现相对简单：比平衡树等数据结构更容易实现
内存效率高：相比树结构有更好的空间利用率
应用广泛：从编程语言到数据库都有应用

设计要点

哈希函数：决定性能的关键，需要分布均匀、计算快速
冲突解决：链地址法和开放寻址法各有优缺点
负载因子控制：平衡时间和空间效率
动态扩容：保持良好的性能特性

高级技术

Robin Hood 散列：优化探测距离分布
一致性哈希：解决分布式系统的负载均衡
布隆过滤器：概率性数据结构，空间高效
并发散列表：支持多线程安全访问

实际应用

编程语言：Python dict、Java HashMap、C++ unordered_map
数据库：索引结构、查询优化
分布式系统：负载均衡、缓存分片
网络协议：路由表、DNS解析

散列表的设计体现了计算机科学中"平均情况优化"的重要思想。虽然最坏情况下性能可能退化，但通过精心设计的哈希函数和冲突解决策略，可以让平均性能达到近乎完美的 O(1)。这种实用主义的设计哲学，正是散列表能够在各种系统中广泛应用的根本原因！

恭喜你完成了数据结构入门教程系列！从链表到散列表，你已经掌握了程序设计的核心工具。继续学习，探索更广阔的算法世界吧！

]]>

数据结构入门教程：跳表数据结构详解与Java实现

lesshash@blog.lesshash.com/ (lesshash) — Tue, 28 Jan 2025 15:45:00 +0800

🏗️ 引言：链表的高速公路

想象一下在一个巨大的城市中开车：如果只有普通道路，你需要在每个路口停下来决定方向；但如果有高速公路，你可以快速跳过很多路口，只在需要的出口下来。**跳表（Skip List）**就是给链表建造的"高速公路系统"！

跳表是一种随机化数据结构，通过在链表上建立多层索引，将链表的查找时间复杂度从 O(n) 降低到 O(log n)。它比平衡二叉树更容易实现，在并发环境下表现更好，是 Redis 等系统的核心数据结构之一。

流程图表

关系流向：

1
2
3
4


A3["1"] → B3["7"]
A2["1"] → B2["4"] → C2["7"] → D2["10"]
A1["1"] → B1["3"] → C1["4"] → D1["6"] → E1["7"] → F1["9"] → G1["10"] → H1["12"]
A0["1"] → B0["2"] → C0["3"] → D0["4"] → E0["5"] → F0["6"] → G0["7"] → H0["8"] → I0["9"] → J0["10"] → K0["11"] → L0["12"]

🏗️ 跳表的基本原理

跳表节点设计

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


/**
 * 跳表节点类
 */
class SkipListNode {
 int value; // 节点值
 SkipListNode[] forward; // 前向指针数组，每层一个指针

 public SkipListNode(int value, int level) {
 this.value = value;
 this.forward = new SkipListNode[level + 1];
 }

 @Override
 public String toString() {
 return "Node(" + value + ", level=" + (forward.length - 1) + ")";
 }
}

跳表核心实现

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246


/**
 * 跳表数据结构实现
 * 时间复杂度：查找、插入、删除均为 O(log n)
 * 空间复杂度：O(n)
 */
public class SkipList {
 private static final int MAX_LEVEL = 16; // 最大层数
 private static final double P = 0.5; // 随机化概率

 private SkipListNode header; // 头节点
 private int level; // 当前最高层数
 private Random random; // 随机数生成器
 private int size; // 元素个数

 /**
 * 构造函数
 */
 public SkipList() {
 this.header = new SkipListNode(Integer.MIN_VALUE, MAX_LEVEL);
 this.level = 0;
 this.random = new Random();
 this.size = 0;

 System.out.println("创建跳表，最大层数: " + MAX_LEVEL);
 }

 /**
 * 随机生成节点层数
 * 使用几何分布：P(level = k) = (1-p)^k * p
 */
 private int randomLevel() {
 int level = 0;
 while (random.nextDouble() < P && level < MAX_LEVEL) {
 level++;
 }
 return level;
 }

 /**
 * 查找元素
 * @param target 目标值
 * @return 是否找到
 */
 public boolean search(int target) {
 SkipListNode current = header;

 System.out.println("\n查找元素: " + target);

 // 从最高层开始向下查找
 for (int i = level; i >= 0; i--) {
 while (current.forward[i] != null && current.forward[i].value < target) {
 current = current.forward[i];
 System.out.println(" 第 " + i + " 层：移动到节点 " + current.value);
 }
 }

 // 移动到第0层的下一个节点
 current = current.forward[0];

 boolean found = current != null && current.value == target;
 System.out.println("查找结果: " + (found ? "找到" : "未找到"));

 return found;
 }

 /**
 * 插入元素
 * @param value 要插入的值
 */
 public void insert(int value) {
 SkipListNode[] update = new SkipListNode[MAX_LEVEL + 1];
 SkipListNode current = header;

 System.out.println("\n插入元素: " + value);

 // 查找插入位置，记录每层的前驱节点
 for (int i = level; i >= 0; i--) {
 while (current.forward[i] != null && current.forward[i].value < value) {
 current = current.forward[i];
 }
 update[i] = current;
 }

 // 移动到第0层的下一个节点
 current = current.forward[0];

 // 如果元素已存在，不重复插入
 if (current != null && current.value == value) {
 System.out.println("元素已存在，不重复插入");
 return;
 }

 // 随机生成新节点的层数
 int newLevel = randomLevel();
 System.out.println("新节点层数: " + newLevel);

 // 如果新节点层数超过当前最高层，更新最高层
 if (newLevel > level) {
 for (int i = level + 1; i <= newLevel; i++) {
 update[i] = header;
 }
 level = newLevel;
 }

 // 创建新节点
 SkipListNode newNode = new SkipListNode(value, newLevel);

 // 更新各层的指针
 for (int i = 0; i <= newLevel; i++) {
 newNode.forward[i] = update[i].forward[i];
 update[i].forward[i] = newNode;
 System.out.println(" 第 " + i + " 层：插入节点");
 }

 size++;
 System.out.println("插入完成，当前大小: " + size);
 }

 /**
 * 删除元素
 * @param target 要删除的值
 * @return 是否删除成功
 */
 public boolean delete(int target) {
 SkipListNode[] update = new SkipListNode[MAX_LEVEL + 1];
 SkipListNode current = header;

 System.out.println("\n删除元素: " + target);

 // 查找要删除的节点，记录每层的前驱节点
 for (int i = level; i >= 0; i--) {
 while (current.forward[i] != null && current.forward[i].value < target) {
 current = current.forward[i];
 }
 update[i] = current;
 }

 // 移动到要删除的节点
 current = current.forward[0];

 // 如果找到目标节点
 if (current != null && current.value == target) {
 // 更新各层的指针
 for (int i = 0; i <= level; i++) {
 if (update[i].forward[i] != current) {
 break;
 }
 update[i].forward[i] = current.forward[i];
 System.out.println(" 第 " + i + " 层：删除节点连接");
 }

 // 如果删除的是最高层的节点，可能需要降低层数
 while (level > 0 && header.forward[level] == null) {
 level--;
 }

 size--;
 System.out.println("删除成功，当前大小: " + size);
 return true;
 }

 System.out.println("未找到要删除的元素");
 return false;
 }

 /**
 * 获取跳表大小
 */
 public int size() {
 return size;
 }

 /**
 * 判断跳表是否为空
 */
 public boolean isEmpty() {
 return size == 0;
 }

 /**
 * 显示跳表结构
 */
 public void display() {
 System.out.println("\n跳表结构 (当前层数: " + level + "):");

 for (int i = level; i >= 0; i--) {
 System.out.print("Level " + i + ": ");
 SkipListNode current = header.forward[i];

 while (current != null) {
 System.out.print(current.value + " -> ");
 current = current.forward[i];
 }
 System.out.println("null");
 }
 System.out.println("总元素数: " + size);
 }

 /**
 * 获取所有元素（有序）
 */
 public List<Integer> toList() {
 List<Integer> result = new ArrayList<>();
 SkipListNode current = header.forward[0];

 while (current != null) {
 result.add(current.value);
 current = current.forward[0];
 }

 return result;
 }

 /**
 * 查找第一个大于等于target的元素
 */
 public Integer ceiling(int target) {
 SkipListNode current = header;

 for (int i = level; i >= 0; i--) {
 while (current.forward[i] != null && current.forward[i].value < target) {
 current = current.forward[i];
 }
 }

 current = current.forward[0];
 return current != null ? current.value : null;
 }

 /**
 * 查找第一个小于target的元素
 */
 public Integer floor(int target) {
 SkipListNode current = header;
 SkipListNode result = null;

 for (int i = level; i >= 0; i--) {
 while (current.forward[i] != null && current.forward[i].value < target) {
 current = current.forward[i];
 result = current;
 }
 }

 return result != null && result != header ? result.value : null;
 }
}

🎯 跳表的高级特性

范围查询实现

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74


/**
 * 跳表范围查询扩展
 */
public class SkipListWithRangeQuery extends SkipList {

 /**
 * 范围查询：返回[min, max]范围内的所有元素
 */
 public List<Integer> rangeQuery(int min, int max) {
 List<Integer> result = new ArrayList<>();

 if (min > max) {
 return result;
 }

 System.out.println("\n范围查询: [" + min + ", " + max + "]");

 // 找到第一个大于等于min的节点
 SkipListNode current = header;
 for (int i = level; i >= 0; i--) {
 while (current.forward[i] != null && current.forward[i].value < min) {
 current = current.forward[i];
 }
 }

 // 从找到的位置开始收集结果
 current = current.forward[0];
 while (current != null && current.value <= max) {
 result.add(current.value);
 current = current.forward[0];
 }

 System.out.println("范围查询结果: " + result);
 return result;
 }

 /**
 * 计算范围内元素个数
 */
 public int countInRange(int min, int max) {
 return rangeQuery(min, max).size();
 }

 /**
 * 获取第k小的元素
 */
 public Integer kthElement(int k) {
 if (k <= 0 || k > size) {
 return null;
 }

 SkipListNode current = header.forward[0];
 for (int i = 1; i < k && current != null; i++) {
 current = current.forward[0];
 }

 return current != null ? current.value : null;
 }

 /**
 * 获取元素的排名（从1开始）
 */
 public int rank(int value) {
 int rank = 1;
 SkipListNode current = header.forward[0];

 while (current != null && current.value < value) {
 rank++;
 current = current.forward[0];
 }

 return current != null && current.value == value ? rank : -1;
 }
}

并发安全的跳表

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94


/**
 * 线程安全的跳表实现
 * 使用读写锁优化并发性能
 */
public class ConcurrentSkipList {
 private final SkipList skipList;
 private final ReadWriteLock lock;
 private final Lock readLock;
 private final Lock writeLock;

 public ConcurrentSkipList() {
 this.skipList = new SkipList();
 this.lock = new ReentrantReadWriteLock();
 this.readLock = lock.readLock();
 this.writeLock = lock.writeLock();
 }

 /**
 * 线程安全的查找
 */
 public boolean search(int target) {
 readLock.lock();
 try {
 return skipList.search(target);
 } finally {
 readLock.unlock();
 }
 }

 /**
 * 线程安全的插入
 */
 public void insert(int value) {
 writeLock.lock();
 try {
 skipList.insert(value);
 } finally {
 writeLock.unlock();
 }
 }

 /**
 * 线程安全的删除
 */
 public boolean delete(int target) {
 writeLock.lock();
 try {
 return skipList.delete(target);
 } finally {
 writeLock.unlock();
 }
 }

 /**
 * 线程安全的大小获取
 */
 public int size() {
 readLock.lock();
 try {
 return skipList.size();
 } finally {
 readLock.unlock();
 }
 }

 /**
 * 批量插入（优化写锁使用）
 */
 public void bulkInsert(int[] values) {
 writeLock.lock();
 try {
 for (int value : values) {
 skipList.insert(value);
 }
 } finally {
 writeLock.unlock();
 }
 }

 /**
 * 线程安全的范围查询
 */
 public List<Integer> rangeQuery(int min, int max) {
 readLock.lock();
 try {
 if (skipList instanceof SkipListWithRangeQuery) {
 return ((SkipListWithRangeQuery) skipList).rangeQuery(min, max);
 }
 return new ArrayList<>();
 } finally {
 readLock.unlock();
 }
 }
}

🔍 跳表的实际应用

Redis中的跳表

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264


/**
 * 模拟Redis中的有序集合（Sorted Set）
 * 基于跳表实现
 */
public class RedisSortedSet {

 /**
 * 有序集合元素
 */
 static class ZSetEntry {
 String member; // 成员名
 double score; // 分数

 ZSetEntry(String member, double score) {
 this.member = member;
 this.score = score;
 }

 @Override
 public String toString() {
 return member + "(" + score + ")";
 }
 }

 /**
 * 跳表节点（支持分数）
 */
 static class ZSkipListNode {
 ZSetEntry entry;
 ZSkipListNode[] forward;

 ZSkipListNode(ZSetEntry entry, int level) {
 this.entry = entry;
 this.forward = new ZSkipListNode[level + 1];
 }

 ZSkipListNode(int level) {
 this.forward = new ZSkipListNode[level + 1];
 }
 }

 private static final int MAX_LEVEL = 32;
 private static final double P = 0.25;

 private ZSkipListNode header;
 private int level;
 private int size;
 private Random random;
 private Map<String, Double> memberToScore; // 成员到分数的映射

 public RedisSortedSet() {
 this.header = new ZSkipListNode(MAX_LEVEL);
 this.level = 0;
 this.size = 0;
 this.random = new Random();
 this.memberToScore = new HashMap<>();
 }

 private int randomLevel() {
 int level = 0;
 while (random.nextDouble() < P && level < MAX_LEVEL) {
 level++;
 }
 return level;
 }

 /**
 * 添加成员到有序集合
 */
 public boolean zadd(String member, double score) {
 // 如果成员已存在，更新分数
 if (memberToScore.containsKey(member)) {
 double oldScore = memberToScore.get(member);
 if (oldScore != score) {
 zrem(member);
 return zaddNew(member, score);
 }
 return false;
 }

 return zaddNew(member, score);
 }

 private boolean zaddNew(String member, double score) {
 ZSkipListNode[] update = new ZSkipListNode[MAX_LEVEL + 1];
 ZSkipListNode current = header;

 // 查找插入位置
 for (int i = level; i >= 0; i--) {
 while (current.forward[i] != null &&
 compareEntry(current.forward[i].entry, score, member) < 0) {
 current = current.forward[i];
 }
 update[i] = current;
 }

 int newLevel = randomLevel();
 if (newLevel > level) {
 for (int i = level + 1; i <= newLevel; i++) {
 update[i] = header;
 }
 level = newLevel;
 }

 ZSkipListNode newNode = new ZSkipListNode(new ZSetEntry(member, score), newLevel);

 for (int i = 0; i <= newLevel; i++) {
 newNode.forward[i] = update[i].forward[i];
 update[i].forward[i] = newNode;
 }

 memberToScore.put(member, score);
 size++;
 return true;
 }

 /**
 * 删除成员
 */
 public boolean zrem(String member) {
 if (!memberToScore.containsKey(member)) {
 return false;
 }

 double score = memberToScore.get(member);
 ZSkipListNode[] update = new ZSkipListNode[MAX_LEVEL + 1];
 ZSkipListNode current = header;

 for (int i = level; i >= 0; i--) {
 while (current.forward[i] != null &&
 compareEntry(current.forward[i].entry, score, member) < 0) {
 current = current.forward[i];
 }
 update[i] = current;
 }

 current = current.forward[0];
 if (current != null && current.entry.member.equals(member)) {
 for (int i = 0; i <= level; i++) {
 if (update[i].forward[i] != current) break;
 update[i].forward[i] = current.forward[i];
 }

 while (level > 0 && header.forward[level] == null) {
 level--;
 }

 memberToScore.remove(member);
 size--;
 return true;
 }

 return false;
 }

 /**
 * 获取成员分数
 */
 public Double zscore(String member) {
 return memberToScore.get(member);
 }

 /**
 * 获取成员排名（从0开始，分数从小到大）
 */
 public int zrank(String member) {
 if (!memberToScore.containsKey(member)) {
 return -1;
 }

 double score = memberToScore.get(member);
 int rank = 0;
 ZSkipListNode current = header.forward[0];

 while (current != null) {
 if (current.entry.member.equals(member)) {
 return rank;
 }
 if (compareEntry(current.entry, score, member) < 0) {
 rank++;
 }
 current = current.forward[0];
 }

 return -1;
 }

 /**
 * 按分数范围查询
 */
 public List<ZSetEntry> zrangeByScore(double min, double max) {
 List<ZSetEntry> result = new ArrayList<>();
 ZSkipListNode current = header.forward[0];

 while (current != null) {
 if (current.entry.score >= min && current.entry.score <= max) {
 result.add(current.entry);
 } else if (current.entry.score > max) {
 break;
 }
 current = current.forward[0];
 }

 return result;
 }

 /**
 * 按排名范围查询
 */
 public List<ZSetEntry> zrange(int start, int stop) {
 List<ZSetEntry> result = new ArrayList<>();
 if (start < 0) start = Math.max(0, size + start);
 if (stop < 0) stop = Math.max(-1, size + stop);

 if (start > stop || start >= size) {
 return result;
 }

 ZSkipListNode current = header.forward[0];
 int index = 0;

 while (current != null && index <= stop) {
 if (index >= start) {
 result.add(current.entry);
 }
 current = current.forward[0];
 index++;
 }

 return result;
 }

 /**
 * 比较两个entry的顺序
 */
 private int compareEntry(ZSetEntry entry, double score, String member) {
 if (entry.score != score) {
 return Double.compare(entry.score, score);
 }
 return entry.member.compareTo(member);
 }

 /**
 * 获取集合大小
 */
 public int zcard() {
 return size;
 }

 /**
 * 显示有序集合
 */
 public void display() {
 System.out.println("有序集合内容 (按分数排序):");
 ZSkipListNode current = header.forward[0];
 int rank = 0;

 while (current != null) {
 System.out.println(" " + rank + ": " + current.entry);
 current = current.forward[0];
 rank++;
 }
 }
}

基于跳表的LRU缓存

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196


/**
 * 基于跳表实现的LRU缓存
 * 结合时间戳和跳表实现O(log n)的LRU
 */
public class SkipListLRUCache<K, V> {

 /**
 * 缓存项
 */
 static class CacheEntry<K, V> {
 K key;
 V value;
 long timestamp;

 CacheEntry(K key, V value, long timestamp) {
 this.key = key;
 this.value = value;
 this.timestamp = timestamp;
 }
 }

 /**
 * 时间戳跳表节点
 */
 static class TimestampNode<K, V> {
 CacheEntry<K, V> entry;
 TimestampNode<K, V>[] forward;

 @SuppressWarnings("unchecked")
 TimestampNode(CacheEntry<K, V> entry, int level) {
 this.entry = entry;
 this.forward = new TimestampNode[level + 1];
 }

 @SuppressWarnings("unchecked")
 TimestampNode(int level) {
 this.forward = new TimestampNode[level + 1];
 }
 }

 private final int capacity;
 private final Map<K, TimestampNode<K, V>> keyToNode;
 private final TimestampNode<K, V> header;
 private int level;
 private int size;
 private final Random random;
 private long currentTime;

 public SkipListLRUCache(int capacity) {
 this.capacity = capacity;
 this.keyToNode = new HashMap<>();
 this.header = new TimestampNode<>(16);
 this.level = 0;
 this.size = 0;
 this.random = new Random();
 this.currentTime = 0;
 }

 /**
 * 获取缓存值
 */
 public V get(K key) {
 TimestampNode<K, V> node = keyToNode.get(key);
 if (node == null) {
 return null;
 }

 // 更新访问时间
 V value = node.entry.value;
 remove(node);
 put(key, value);

 return value;
 }

 /**
 * 放入缓存
 */
 public void put(K key, V value) {
 // 如果key已存在，删除旧节点
 if (keyToNode.containsKey(key)) {
 TimestampNode<K, V> oldNode = keyToNode.get(key);
 remove(oldNode);
 }

 // 如果达到容量限制，删除最旧的节点
 if (size >= capacity) {
 removeOldest();
 }

 // 插入新节点
 CacheEntry<K, V> entry = new CacheEntry<>(key, value, ++currentTime);
 insertNode(entry);
 }

 /**
 * 插入新节点到跳表
 */
 private void insertNode(CacheEntry<K, V> entry) {
 TimestampNode<K, V>[] update = new TimestampNode[17];
 TimestampNode<K, V> current = header;

 // 查找插入位置
 for (int i = level; i >= 0; i--) {
 while (current.forward[i] != null &&
 current.forward[i].entry.timestamp < entry.timestamp) {
 current = current.forward[i];
 }
 update[i] = current;
 }

 int newLevel = randomLevel();
 if (newLevel > level) {
 for (int i = level + 1; i <= newLevel; i++) {
 update[i] = header;
 }
 level = newLevel;
 }

 TimestampNode<K, V> newNode = new TimestampNode<>(entry, newLevel);

 for (int i = 0; i <= newLevel; i++) {
 newNode.forward[i] = update[i].forward[i];
 update[i].forward[i] = newNode;
 }

 keyToNode.put(entry.key, newNode);
 size++;
 }

 /**
 * 删除最旧的节点
 */
 private void removeOldest() {
 TimestampNode<K, V> oldest = header.forward[0];
 if (oldest != null) {
 remove(oldest);
 }
 }

 /**
 * 删除指定节点
 */
 private void remove(TimestampNode<K, V> nodeToRemove) {
 TimestampNode<K, V>[] update = new TimestampNode[17];
 TimestampNode<K, V> current = header;

 // 查找要删除的节点
 for (int i = level; i >= 0; i--) {
 while (current.forward[i] != null &&
 current.forward[i].entry.timestamp < nodeToRemove.entry.timestamp) {
 current = current.forward[i];
 }
 update[i] = current;
 }

 current = current.forward[0];
 if (current == nodeToRemove) {
 for (int i = 0; i <= level; i++) {
 if (update[i].forward[i] != current) break;
 update[i].forward[i] = current.forward[i];
 }

 while (level > 0 && header.forward[level] == null) {
 level--;
 }

 keyToNode.remove(nodeToRemove.entry.key);
 size--;
 }
 }

 private int randomLevel() {
 int level = 0;
 while (random.nextDouble() < 0.5 && level < 16) {
 level++;
 }
 return level;
 }

 /**
 * 显示缓存状态
 */
 public void display() {
 System.out.println("LRU缓存状态 (按时间戳排序):");
 TimestampNode<K, V> current = header.forward[0];

 while (current != null) {
 System.out.println(" " + current.entry.key + " -> " +
 current.entry.value + " (time: " +
 current.entry.timestamp + ")");
 current = current.forward[0];
 }
 System.out.println("缓存大小: " + size + "/" + capacity);
 }
}

📊 性能分析与优化

跳表性能测试

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185


/**
 * 跳表性能分析工具
 */
public class SkipListPerformanceAnalyzer {

 /**
 * 比较跳表与其他数据结构的性能
 */
 public static void compareDataStructures() {
 int[] sizes = {1000, 10000, 100000, 1000000};

 System.out.println("数据结构性能对比:");
 System.out.println("操作\\数据结构\t跳表\t\tTreeSet\t\tLinkedList");
 System.out.println("-".repeat(60));

 for (int size : sizes) {
 System.out.println("\n数据规模: " + size);

 // 测试数据
 int[] testData = generateRandomData(size);
 int[] searchData = generateRandomData(1000);

 // 跳表测试
 long skipListTime = testSkipList(testData, searchData);

 // TreeSet测试
 long treeSetTime = testTreeSet(testData, searchData);

 // LinkedList测试（只测试小规模数据）
 long linkedListTime = size <= 10000 ? testLinkedList(testData, searchData) : -1;

 System.out.printf("插入+查找\t%.2fms\t\t%.2fms\t\t%s%n",
 skipListTime / 1_000_000.0,
 treeSetTime / 1_000_000.0,
 linkedListTime == -1 ? "太慢" : String.format("%.2fms", linkedListTime / 1_000_000.0));
 }
 }

 private static long testSkipList(int[] insertData, int[] searchData) {
 SkipList skipList = new SkipList();

 long startTime = System.nanoTime();

 // 插入操作
 for (int value : insertData) {
 skipList.insert(value);
 }

 // 查找操作
 for (int value : searchData) {
 skipList.search(value);
 }

 return System.nanoTime() - startTime;
 }

 private static long testTreeSet(int[] insertData, int[] searchData) {
 TreeSet<Integer> treeSet = new TreeSet<>();

 long startTime = System.nanoTime();

 // 插入操作
 for (int value : insertData) {
 treeSet.add(value);
 }

 // 查找操作
 for (int value : searchData) {
 treeSet.contains(value);
 }

 return System.nanoTime() - startTime;
 }

 private static long testLinkedList(int[] insertData, int[] searchData) {
 LinkedList<Integer> linkedList = new LinkedList<>();

 long startTime = System.nanoTime();

 // 插入操作（保持有序）
 for (int value : insertData) {
 insertSorted(linkedList, value);
 }

 // 查找操作
 for (int value : searchData) {
 linkedList.contains(value);
 }

 return System.nanoTime() - startTime;
 }

 private static void insertSorted(LinkedList<Integer> list, int value) {
 int index = 0;
 for (Integer item : list) {
 if (item > value) {
 break;
 }
 index++;
 }
 list.add(index, value);
 }

 private static int[] generateRandomData(int size) {
 Random random = new Random(42);
 Set<Integer> uniqueValues = new HashSet<>();

 while (uniqueValues.size() < size) {
 uniqueValues.add(random.nextInt(size * 10));
 }

 return uniqueValues.stream().mapToInt(i -> i).toArray();
 }

 /**
 * 分析跳表的层数分布
 */
 public static void analyzeLevelDistribution() {
 SkipList skipList = new SkipList();
 int[] testData = generateRandomData(10000);

 Map<Integer, Integer> levelCount = new HashMap<>();

 // 插入数据并统计层数分布
 for (int value : testData) {
 skipList.insert(value);
 }

 // 遍历跳表统计每层节点数
 System.out.println("跳表层数分布分析:");
 skipList.display();
 }

 /**
 * 测试并发性能
 */
 public static void testConcurrentPerformance() {
 ConcurrentSkipList concurrentSkipList = new ConcurrentSkipList();
 int numThreads = 4;
 int operationsPerThread = 10000;

 ExecutorService executor = Executors.newFixedThreadPool(numThreads);
 CountDownLatch latch = new CountDownLatch(numThreads);

 long startTime = System.nanoTime();

 for (int i = 0; i < numThreads; i++) {
 final int threadId = i;
 executor.submit(() -> {
 try {
 Random random = new Random(threadId);

 for (int j = 0; j < operationsPerThread; j++) {
 int value = random.nextInt(100000);

 if (j % 3 == 0) {
 concurrentSkipList.insert(value);
 } else if (j % 3 == 1) {
 concurrentSkipList.search(value);
 } else {
 concurrentSkipList.delete(value);
 }
 }
 } finally {
 latch.countDown();
 }
 });
 }

 try {
 latch.await();
 long endTime = System.nanoTime();

 System.out.printf("并发测试完成: %d线程, 每线程%d操作, 总耗时: %.2fms%n",
 numThreads, operationsPerThread,
 (endTime - startTime) / 1_000_000.0);
 System.out.println("最终大小: " + concurrentSkipList.size());

 } catch (InterruptedException e) {
 Thread.currentThread().interrupt();
 } finally {
 executor.shutdown();
 }
 }
}

🧪 完整测试示例

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98


/**
 * 跳表数据结构综合测试
 */
public class SkipListTest {
 public static void main(String[] args) {
 System.out.println("=== 跳表数据结构综合测试 ===");

 testBasicOperations();
 testRangeQueries();
 testRedisSortedSet();
 testLRUCache();
 testPerformance();
 }

 private static void testBasicOperations() {
 System.out.println("\n1. 基本操作测试:");

 SkipList skipList = new SkipList();

 // 插入测试
 int[] values = {3, 6, 7, 9, 12, 17, 19, 21, 25, 26};
 for (int value : values) {
 skipList.insert(value);
 }

 skipList.display();

 // 查找测试
 System.out.println("\n查找测试:");
 System.out.println("查找7: " + skipList.search(7));
 System.out.println("查找15: " + skipList.search(15));

 // 删除测试
 System.out.println("\n删除测试:");
 skipList.delete(7);
 skipList.delete(25);
 skipList.display();
 }

 private static void testRangeQueries() {
 System.out.println("\n2. 范围查询测试:");

 SkipListWithRangeQuery skipList = new SkipListWithRangeQuery();

 for (int i = 1; i <= 20; i++) {
 skipList.insert(i);
 }

 System.out.println("范围[5, 15]: " + skipList.rangeQuery(5, 15));
 System.out.println("第8小元素: " + skipList.kthElement(8));
 System.out.println("元素12的排名: " + skipList.rank(12));
 }

 private static void testRedisSortedSet() {
 System.out.println("\n3. Redis有序集合测试:");

 RedisSortedSet zset = new RedisSortedSet();

 // 添加成员
 zset.zadd("Alice", 85.5);
 zset.zadd("Bob", 92.0);
 zset.zadd("Charlie", 78.5);
 zset.zadd("Diana", 96.5);
 zset.zadd("Eve", 88.0);

 zset.display();

 System.out.println("\nBob的分数: " + zset.zscore("Bob"));
 System.out.println("Charlie的排名: " + zset.zrank("Charlie"));
 System.out.println("分数90-100范围: " + zset.zrangeByScore(90, 100));
 System.out.println("排名0-2: " + zset.zrange(0, 2));
 }

 private static void testLRUCache() {
 System.out.println("\n4. LRU缓存测试:");

 SkipListLRUCache<String, String> cache = new SkipListLRUCache<>(3);

 cache.put("A", "Value A");
 cache.put("B", "Value B");
 cache.put("C", "Value C");
 cache.display();

 System.out.println("\n访问A: " + cache.get("A"));
 cache.display();

 System.out.println("\n添加D (超出容量):");
 cache.put("D", "Value D");
 cache.display();
 }

 private static void testPerformance() {
 System.out.println("\n5. 性能测试:");

 SkipListPerformanceAnalyzer.compareDataStructures();
 SkipListPerformanceAnalyzer.testConcurrentPerformance();
 }
}

🎯 总结

跳表是一种优雅的概率数据结构，完美平衡了性能和实现复杂度：

核心优势

性能优秀：O(log n) 的查找、插入、删除时间复杂度
实现简单：比平衡二叉树更容易实现和维护
并发友好：天然支持无锁并发操作
内存局部性好：相比树结构有更好的缓存性能

关键特性

随机化层数：通过概率分布构建多层索引
有序性：天然维护元素的有序性
范围查询：高效支持范围操作
动态性：支持动态插入和删除

实际应用

Redis：有序集合（Sorted Set）的底层实现
数据库：索引结构的一种选择
缓存系统：LRU缓存的高效实现
搜索引擎：倒排索引的组织结构

设计思想

跳表的核心思想是"空间换时间"和"概率平衡"。它用额外的空间建立多层索引，用随机化避免最坏情况，实现了简单而高效的有序数据结构。

跳表证明了有时候最优雅的解决方案不一定是最复杂的——通过巧妙的设计和概率分析，我们可以用相对简单的方法达到很好的效果！

下一篇：《数据结构入门教程：散列表数据结构详解与Java实现》

]]>

数据结构入门教程：二分查找算法详解与Java实现

lesshash@blog.lesshash.com/ (lesshash) — Tue, 28 Jan 2025 15:30:00 +0800

🎯 引言：分而治之的查找艺术

想象一下在字典中查找单词的过程：你不会从第一页开始逐页翻阅，而是先翻到中间，根据字母顺序决定往前还是往后，然后继续对剩余部分重复这个过程。这就是二分查找的基本思想！

**二分查找（Binary Search）**是一种在有序数组中查找特定元素的高效算法。它每次将搜索范围缩小一半，时间复杂度仅为 O(log n)，是查找算法中的经典之作。

流程图表

关系流向：

1
2
3
4


A["有序数组 [1,3,5,7,9,11,13,15]"] → B["查找目标: 7"]
B → C["中间位置: arr[3] = 7"]
C → D["比较结果"]
D → E["找到目标!"]

🔍 二分查找基础实现

经典二分查找

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91


/**
 * 二分查找基础实现
 * 时间复杂度：O(log n)
 * 空间复杂度：O(1)
 */
public class BinarySearch {

 /**
 * 基础二分查找（迭代版本）
 * @param arr 有序数组
 * @param target 目标值
 * @return 目标值的索引，未找到返回-1
 */
 public static int binarySearchIterative(int[] arr, int target) {
 int left = 0;
 int right = arr.length - 1;

 System.out.println("查找目标: " + target);
 System.out.println("数组: " + Arrays.toString(arr));

 while (left <= right) {
 int mid = left + (right - left) / 2; // 防止溢出

 System.out.printf("搜索范围: [%d, %d], 中间位置: %d, 中间值: %d%n",
 left, right, mid, arr[mid]);

 if (arr[mid] == target) {
 System.out.println("找到目标，位置: " + mid);
 return mid;
 } else if (arr[mid] < target) {
 left = mid + 1;
 System.out.println("目标在右半部分");
 } else {
 right = mid - 1;
 System.out.println("目标在左半部分");
 }
 }

 System.out.println("未找到目标");
 return -1;
 }

 /**
 * 递归版本的二分查找
 * @param arr 有序数组
 * @param target 目标值
 * @param left 左边界
 * @param right 右边界
 * @return 目标值的索引，未找到返回-1
 */
 public static int binarySearchRecursive(int[] arr, int target, int left, int right) {
 if (left > right) {
 return -1; // 未找到
 }

 int mid = left + (right - left) / 2;
 System.out.printf("递归查找: [%d, %d], 中间位置: %d, 中间值: %d%n",
 left, right, mid, arr[mid]);

 if (arr[mid] == target) {
 return mid;
 } else if (arr[mid] < target) {
 return binarySearchRecursive(arr, target, mid + 1, right);
 } else {
 return binarySearchRecursive(arr, target, left, mid - 1);
 }
 }

 /**
 * 二分查找的边界处理模板
 * 这是最不容易出错的写法
 */
 public static int binarySearchTemplate(int[] arr, int target) {
 int left = 0;
 int right = arr.length; // 注意：这里是length，不是length-1

 while (left < right) { // 注意：这里是<，不是<=
 int mid = left + (right - left) / 2;

 if (arr[mid] == target) {
 return mid;
 } else if (arr[mid] < target) {
 left = mid + 1;
 } else {
 right = mid; // 注意：这里是mid，不是mid-1
 }
 }

 return -1;
 }
}

二分查找的变种

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174


/**
 * 二分查找的各种变种
 * 处理重复元素和边界情况
 */
public class BinarySearchVariants {

 /**
 * 查找第一个等于目标值的位置
 * 适用于有重复元素的有序数组
 */
 public static int findFirst(int[] arr, int target) {
 int left = 0;
 int right = arr.length - 1;
 int result = -1;

 while (left <= right) {
 int mid = left + (right - left) / 2;

 if (arr[mid] == target) {
 result = mid;
 right = mid - 1; // 继续在左半部分查找
 } else if (arr[mid] < target) {
 left = mid + 1;
 } else {
 right = mid - 1;
 }
 }

 return result;
 }

 /**
 * 查找最后一个等于目标值的位置
 */
 public static int findLast(int[] arr, int target) {
 int left = 0;
 int right = arr.length - 1;
 int result = -1;

 while (left <= right) {
 int mid = left + (right - left) / 2;

 if (arr[mid] == target) {
 result = mid;
 left = mid + 1; // 继续在右半部分查找
 } else if (arr[mid] < target) {
 left = mid + 1;
 } else {
 right = mid - 1;
 }
 }

 return result;
 }

 /**
 * 查找第一个大于等于目标值的位置（下界）
 */
 public static int lowerBound(int[] arr, int target) {
 int left = 0;
 int right = arr.length;

 while (left < right) {
 int mid = left + (right - left) / 2;

 if (arr[mid] < target) {
 left = mid + 1;
 } else {
 right = mid;
 }
 }

 return left;
 }

 /**
 * 查找第一个大于目标值的位置（上界）
 */
 public static int upperBound(int[] arr, int target) {
 int left = 0;
 int right = arr.length;

 while (left < right) {
 int mid = left + (right - left) / 2;

 if (arr[mid] <= target) {
 left = mid + 1;
 } else {
 right = mid;
 }
 }

 return left;
 }

 /**
 * 查找目标值的范围（第一个和最后一个位置）
 */
 public static int[] searchRange(int[] arr, int target) {
 int first = findFirst(arr, target);
 if (first == -1) {
 return new int[]{-1, -1};
 }

 int last = findLast(arr, target);
 return new int[]{first, last};
 }

 /**
 * 查找插入位置
 * 在有序数组中找到插入target后仍保持有序的位置
 */
 public static int searchInsert(int[] arr, int target) {
 return lowerBound(arr, target);
 }

 /**
 * 在旋转排序数组中查找元素
 */
 public static int searchInRotatedArray(int[] nums, int target) {
 int left = 0;
 int right = nums.length - 1;

 while (left <= right) {
 int mid = left + (right - left) / 2;

 if (nums[mid] == target) {
 return mid;
 }

 // 判断哪一部分是有序的
 if (nums[left] <= nums[mid]) {
 // 左半部分有序
 if (nums[left] <= target && target < nums[mid]) {
 right = mid - 1;
 } else {
 left = mid + 1;
 }
 } else {
 // 右半部分有序
 if (nums[mid] < target && target <= nums[right]) {
 left = mid + 1;
 } else {
 right = mid - 1;
 }
 }
 }

 return -1;
 }

 /**
 * 查找峰值元素
 * 峰值元素是指其值大于左右相邻值的元素
 */
 public static int findPeakElement(int[] nums) {
 int left = 0;
 int right = nums.length - 1;

 while (left < right) {
 int mid = left + (right - left) / 2;

 if (nums[mid] > nums[mid + 1]) {
 // 峰值在左侧（包括mid）
 right = mid;
 } else {
 // 峰值在右侧
 left = mid + 1;
 }
 }

 return left;
 }
}

🎮 二分答案

二分答案是二分查找的高级应用，用于求解最优化问题。

二分答案模板

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178


/**
 * 二分答案算法实现
 * 用于求解最优化问题
 */
public class BinaryAnswer {

 /**
 * 二分答案通用模板
 * @param left 答案的最小可能值
 * @param right 答案的最大可能值
 * @param checker 检查函数，判断某个值是否满足条件
 * @param findMinimum 是否查找最小值（true）还是最大值（false）
 * @return 最优答案
 */
 public static int binaryAnswer(int left, int right,
 IntPredicate checker, boolean findMinimum) {
 while (left < right) {
 int mid = left + (right - left) / 2;

 if (findMinimum) {
 // 查找最小的满足条件的值
 if (checker.test(mid)) {
 right = mid;
 } else {
 left = mid + 1;
 }
 } else {
 // 查找最大的满足条件的值
 if (checker.test(mid)) {
 left = mid + 1;
 } else {
 right = mid;
 }
 }
 }

 return findMinimum ? left : left - 1;
 }

 /**
 * 案例1：分割数组的最大值
 * 给定一个数组和分割数m，将数组分成m个非空连续子数组，
 * 使得这m个子数组各自和的最大值最小
 */
 public static int splitArray(int[] nums, int m) {
 int left = Arrays.stream(nums).max().orElse(0); // 最小可能的最大值
 int right = Arrays.stream(nums).sum(); // 最大可能的最大值

 System.out.println("分割数组问题:");
 System.out.println("数组: " + Arrays.toString(nums));
 System.out.println("分割数: " + m);
 System.out.println("搜索范围: [" + left + ", " + right + "]");

 while (left < right) {
 int mid = left + (right - left) / 2;

 if (canSplit(nums, m, mid)) {
 System.out.println("最大和 " + mid + " 可以分割");
 right = mid;
 } else {
 System.out.println("最大和 " + mid + " 无法分割");
 left = mid + 1;
 }
 }

 System.out.println("最小的最大子数组和: " + left);
 return left;
 }

 /**
 * 检查是否可以将数组分成m个子数组，且每个子数组和不超过maxSum
 */
 private static boolean canSplit(int[] nums, int m, int maxSum) {
 int count = 1; // 至少需要一个子数组
 int currentSum = 0;

 for (int num : nums) {
 if (currentSum + num > maxSum) {
 count++;
 currentSum = num;
 if (count > m) {
 return false;
 }
 } else {
 currentSum += num;
 }
 }

 return true;
 }

 /**
 * 案例2：第K小的距离对
 * 给定整数数组，返回所有数对的第K小距离
 */
 public static int smallestDistancePair(int[] nums, int k) {
 Arrays.sort(nums);

 int left = 0;
 int right = nums[nums.length - 1] - nums[0];

 System.out.println("第K小距离对问题:");
 System.out.println("数组: " + Arrays.toString(nums));
 System.out.println("K: " + k);

 while (left < right) {
 int mid = left + (right - left) / 2;
 int count = countPairsWithDistanceLEQ(nums, mid);

 System.out.println("距离 " + mid + " 的对数: " + count);

 if (count < k) {
 left = mid + 1;
 } else {
 right = mid;
 }
 }

 return left;
 }

 /**
 * 计算距离小于等于target的数对个数
 */
 private static int countPairsWithDistanceLEQ(int[] nums, int target) {
 int count = 0;
 int left = 0;

 for (int right = 1; right < nums.length; right++) {
 while (nums[right] - nums[left] > target) {
 left++;
 }
 count += right - left;
 }

 return count;
 }

 /**
 * 案例3：爱吃香蕉的珂珂
 * 珂珂有H小时吃香蕉，找到她可以在H小时内吃掉所有香蕉的最小速度
 */
 public static int minEatingSpeed(int[] piles, int h) {
 int left = 1;
 int right = Arrays.stream(piles).max().orElse(1);

 System.out.println("爱吃香蕉的珂珂问题:");
 System.out.println("香蕉堆: " + Arrays.toString(piles));
 System.out.println("时间限制: " + h + " 小时");

 while (left < right) {
 int mid = left + (right - left) / 2;
 int timeNeeded = calculateTimeNeeded(piles, mid);

 System.out.println("速度 " + mid + " 需要时间: " + timeNeeded);

 if (timeNeeded <= h) {
 right = mid;
 } else {
 left = mid + 1;
 }
 }

 System.out.println("最小吃香蕉速度: " + left);
 return left;
 }

 /**
 * 计算以指定速度吃香蕉需要的时间
 */
 private static int calculateTimeNeeded(int[] piles, int speed) {
 int time = 0;
 for (int pile : piles) {
 time += (pile + speed - 1) / speed; // 向上取整
 }
 return time;
 }
}

🔍 二分查找在数据结构中的应用

二分查找树（Binary Search Tree）

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94


/**
 * 二分查找树实现
 * 结合二分查找思想的树结构
 */
public class BinarySearchTree {
 private TreeNode root;

 private static class TreeNode {
 int val;
 TreeNode left;
 TreeNode right;

 TreeNode(int val) {
 this.val = val;
 }
 }

 /**
 * 插入节点
 */
 public void insert(int val) {
 root = insertHelper(root, val);
 }

 private TreeNode insertHelper(TreeNode node, int val) {
 if (node == null) {
 return new TreeNode(val);
 }

 if (val < node.val) {
 node.left = insertHelper(node.left, val);
 } else if (val > node.val) {
 node.right = insertHelper(node.right, val);
 }

 return node;
 }

 /**
 * 查找节点（二分查找思想）
 */
 public boolean search(int val) {
 return searchHelper(root, val);
 }

 private boolean searchHelper(TreeNode node, int val) {
 if (node == null) {
 return false;
 }

 if (val == node.val) {
 return true;
 } else if (val < node.val) {
 return searchHelper(node.left, val);
 } else {
 return searchHelper(node.right, val);
 }
 }

 /**
 * 找到第K小的元素
 */
 public int kthSmallest(int k) {
 List<Integer> result = new ArrayList<>();
 inorderTraversal(root, result);
 return result.get(k - 1);
 }

 private void inorderTraversal(TreeNode node, List<Integer> result) {
 if (node == null) return;

 inorderTraversal(node.left, result);
 result.add(node.val);
 inorderTraversal(node.right, result);
 }

 /**
 * 验证是否为有效的二分查找树
 */
 public boolean isValidBST() {
 return isValidBSTHelper(root, Long.MIN_VALUE, Long.MAX_VALUE);
 }

 private boolean isValidBSTHelper(TreeNode node, long minVal, long maxVal) {
 if (node == null) return true;

 if (node.val <= minVal || node.val >= maxVal) {
 return false;
 }

 return isValidBSTHelper(node.left, minVal, node.val) &&
 isValidBSTHelper(node.right, node.val, maxVal);
 }
}

有序集合的二分查找

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83


/**
 * 基于有序数组的集合实现
 * 使用二分查找优化操作
 */
public class OrderedSet {
 private List<Integer> data;

 public OrderedSet() {
 this.data = new ArrayList<>();
 }

 /**
 * 添加元素（保持有序）
 */
 public void add(int val) {
 int pos = BinarySearchVariants.lowerBound(
 data.stream().mapToInt(i -> i).toArray(), val);

 if (pos < data.size() && data.get(pos) == val) {
 return; // 元素已存在
 }

 data.add(pos, val);
 }

 /**
 * 删除元素
 */
 public boolean remove(int val) {
 int pos = BinarySearch.binarySearchIterative(
 data.stream().mapToInt(i -> i).toArray(), val);

 if (pos == -1) {
 return false;
 }

 data.remove(pos);
 return true;
 }

 /**
 * 查找元素
 */
 public boolean contains(int val) {
 return BinarySearch.binarySearchIterative(
 data.stream().mapToInt(i -> i).toArray(), val) != -1;
 }

 /**
 * 获取小于val的最大元素
 */
 public Integer lower(int val) {
 int pos = BinarySearchVariants.lowerBound(
 data.stream().mapToInt(i -> i).toArray(), val);

 return pos > 0 ? data.get(pos - 1) : null;
 }

 /**
 * 获取大于val的最小元素
 */
 public Integer higher(int val) {
 int pos = BinarySearchVariants.upperBound(
 data.stream().mapToInt(i -> i).toArray(), val);

 return pos < data.size() ? data.get(pos) : null;
 }

 /**
 * 获取范围内的元素
 */
 public List<Integer> range(int fromVal, int toVal) {
 int[] arr = data.stream().mapToInt(i -> i).toArray();
 int startPos = BinarySearchVariants.lowerBound(arr, fromVal);
 int endPos = BinarySearchVariants.upperBound(arr, toVal);

 return data.subList(startPos, endPos);
 }

 public void display() {
 System.out.println("OrderedSet: " + data);
 }
}

🎯 二分查找的优化技巧

缓存友好的二分查找

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97


/**
 * 缓存友好的二分查找优化
 */
public class CacheFriendlyBinarySearch {

 /**
 * 分块二分查找
 * 先用二分查找定位块，再在块内线性搜索
 */
 public static int blockBinarySearch(int[] arr, int target) {
 int blockSize = (int) Math.sqrt(arr.length);
 int numBlocks = (arr.length + blockSize - 1) / blockSize;

 // 二分查找定位块
 int left = 0, right = numBlocks - 1;
 int targetBlock = -1;

 while (left <= right) {
 int mid = left + (right - left) / 2;
 int blockStart = mid * blockSize;
 int blockEnd = Math.min(blockStart + blockSize - 1, arr.length - 1);

 if (arr[blockStart] <= target && target <= arr[blockEnd]) {
 targetBlock = mid;
 break;
 } else if (target < arr[blockStart]) {
 right = mid - 1;
 } else {
 left = mid + 1;
 }
 }

 if (targetBlock == -1) return -1;

 // 在目标块内线性搜索
 int blockStart = targetBlock * blockSize;
 int blockEnd = Math.min(blockStart + blockSize, arr.length);

 for (int i = blockStart; i < blockEnd; i++) {
 if (arr[i] == target) {
 return i;
 }
 }

 return -1;
 }

 /**
 * 插值搜索（适用于均匀分布的数据）
 * 基于二分查找的改进
 */
 public static int interpolationSearch(int[] arr, int target) {
 int left = 0;
 int right = arr.length - 1;

 while (left <= right && target >= arr[left] && target <= arr[right]) {
 if (left == right) {
 return arr[left] == target ? left : -1;
 }

 // 使用插值公式计算位置
 int pos = left + ((target - arr[left]) * (right - left)) /
 (arr[right] - arr[left]);

 if (arr[pos] == target) {
 return pos;
 } else if (arr[pos] < target) {
 left = pos + 1;
 } else {
 right = pos - 1;
 }
 }

 return -1;
 }

 /**
 * 指数搜索（适用于无界数组）
 */
 public static int exponentialSearch(int[] arr, int target) {
 if (arr[0] == target) {
 return 0;
 }

 // 找到搜索范围
 int bound = 1;
 while (bound < arr.length && arr[bound] < target) {
 bound *= 2;
 }

 // 在范围内进行二分查找
 return BinarySearch.binarySearchIterative(
 Arrays.copyOfRange(arr, bound / 2, Math.min(bound + 1, arr.length)),
 target
 ) + bound / 2;
 }
}

🧪 完整测试示例

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130


/**
 * 二分查找算法综合测试
 */
public class BinarySearchTest {
 public static void main(String[] args) {
 System.out.println("=== 二分查找算法综合测试 ===");

 testBasicBinarySearch();
 testBinarySearchVariants();
 testBinaryAnswer();
 testBinarySearchTree();
 testOrderedSet();
 testOptimizations();
 }

 private static void testBasicBinarySearch() {
 System.out.println("\n1. 基础二分查找测试:");

 int[] arr = {1, 3, 5, 7, 9, 11, 13, 15, 17, 19};
 int target = 7;

 System.out.println("迭代版本:");
 int result1 = BinarySearch.binarySearchIterative(arr, target);

 System.out.println("\n递归版本:");
 int result2 = BinarySearch.binarySearchRecursive(arr, target, 0, arr.length - 1);

 System.out.println("结果: " + result1 + ", " + result2);
 }

 private static void testBinarySearchVariants() {
 System.out.println("\n2. 二分查找变种测试:");

 int[] arr = {1, 2, 2, 2, 3, 4, 4, 5, 5, 5, 6};
 int target = 2;

 System.out.println("数组: " + Arrays.toString(arr));
 System.out.println("目标: " + target);

 int first = BinarySearchVariants.findFirst(arr, target);
 int last = BinarySearchVariants.findLast(arr, target);
 int lower = BinarySearchVariants.lowerBound(arr, target);
 int upper = BinarySearchVariants.upperBound(arr, target);

 System.out.println("第一个位置: " + first);
 System.out.println("最后位置: " + last);
 System.out.println("下界: " + lower);
 System.out.println("上界: " + upper);

 // 测试旋转数组
 int[] rotated = {4, 5, 6, 7, 0, 1, 2};
 int rotatedResult = BinarySearchVariants.searchInRotatedArray(rotated, 0);
 System.out.println("旋转数组中查找0: " + rotatedResult);
 }

 private static void testBinaryAnswer() {
 System.out.println("\n3. 二分答案测试:");

 // 分割数组测试
 int[] nums = {7, 2, 5, 10, 8};
 int m = 2;
 int result = BinaryAnswer.splitArray(nums, m);

 // 香蕉问题测试
 int[] piles = {3, 6, 7, 11};
 int h = 8;
 int speed = BinaryAnswer.minEatingSpeed(piles, h);
 }

 private static void testBinarySearchTree() {
 System.out.println("\n4. 二分查找树测试:");

 BinarySearchTree bst = new BinarySearchTree();
 int[] values = {5, 3, 7, 2, 4, 6, 8};

 for (int val : values) {
 bst.insert(val);
 }

 System.out.println("查找5: " + bst.search(5));
 System.out.println("查找10: " + bst.search(10));
 System.out.println("第3小元素: " + bst.kthSmallest(3));
 System.out.println("是否为有效BST: " + bst.isValidBST());
 }

 private static void testOrderedSet() {
 System.out.println("\n5. 有序集合测试:");

 OrderedSet set = new OrderedSet();
 int[] values = {5, 2, 8, 1, 9, 3};

 for (int val : values) {
 set.add(val);
 }

 set.display();
 System.out.println("包含5: " + set.contains(5));
 System.out.println("小于5的最大元素: " + set.lower(5));
 System.out.println("大于5的最小元素: " + set.higher(5));
 System.out.println("范围[3,7]: " + set.range(3, 7));
 }

 private static void testOptimizations() {
 System.out.println("\n6. 优化技术测试:");

 int[] largeArr = new int[1000000];
 for (int i = 0; i < largeArr.length; i++) {
 largeArr[i] = i * 2;
 }

 int target = 999998;

 // 比较不同搜索算法的性能
 long start = System.nanoTime();
 int result1 = BinarySearch.binarySearchIterative(largeArr, target);
 long time1 = System.nanoTime() - start;

 start = System.nanoTime();
 int result2 = CacheFriendlyBinarySearch.blockBinarySearch(largeArr, target);
 long time2 = System.nanoTime() - start;

 start = System.nanoTime();
 int result3 = CacheFriendlyBinarySearch.interpolationSearch(largeArr, target);
 long time3 = System.nanoTime() - start;

 System.out.println("普通二分查找: " + time1 / 1000.0 + " μs, 结果: " + result1);
 System.out.println("分块二分查找: " + time2 / 1000.0 + " μs, 结果: " + result2);
 System.out.println("插值搜索: " + time3 / 1000.0 + " μs, 结果: " + result3);
 }
}

🎯 总结

二分查找是一种优雅而强大的算法，体现了"分而治之"的编程思想：

核心特点

效率高：O(log n) 时间复杂度，显著优于线性查找
适用范围广：不仅用于数组查找，还有很多变种应用
思想简单：每次排除一半搜索空间
边界处理重要：正确处理边界是关键

应用场景

有序数组查找：最基础的应用
二分答案：求解最优化问题
数据结构：BST、有序集合等
算法优化：降低时间复杂度

实现要点

防止整数溢出：使用 left + (right - left) / 2
边界条件：明确开闭区间的定义
循环不变式：保持搜索区间的性质
变种应用：根据具体问题调整判断条件

二分查找不仅是一个基础算法，更是一种重要的问题解决思路。掌握二分查找的各种变种和应用，将极大提升你解决复杂问题的能力！

下一篇：《数据结构入门教程：跳表数据结构详解与Java实现》

]]>