Go 的 http 服务器,同时下载的连接不能超过两个吗?

2022-06-10 11:12:08 +08:00
 daokedao

一个最简单的 http 服务器:

func main() {
	fs := http.FileServer( http.Dir("./public"))
	http.Handle("/", fs)
	http.ListenAndServe(":8888", nil)
}

用 curl 测试几个连接同时下载:

curl --limit-rate 1k http://1.2.3.4:9999/file.zip --output NUL

结果是,只有第一个和第二个连接正常下载,其他的连接处于停滞状态,只有前面的连接下载完了,后面的才能接着下载。

如果把服务器换成 nginx 则多个连接都可以同时下载。

以上都是在 Windows 11 下进行的 Go 的版本是 1.18.1

4368 次点击
所在节点    Go 编程语言
38 条回复
Kisesy
2022-06-11 18:20:14 +08:00
试了一下还真是,第一个和第二个正常下载,第三个就停住不动,如果结束掉其中一个,第三个就正常下载了
daokedao
2022-06-11 18:22:59 +08:00
@Kisesy 终于能复现了 🤣🤣🤣
Frankcox
2022-06-11 18:23:57 +08:00
@daokedao 我刚刚又试了下,还是没有出现你的问题,同时下载四个 10G 左右的视频。两个视频速度 200mb/s 多,两个视频 50mb/s 左右。其中只有一个视频初始下载速度大概 100B/s 持续了 3 秒左右就正常了。
daokedao
2022-06-11 18:33:29 +08:00
@Frankcox 可我这怎么都不行
Kisesy
2022-06-11 19:02:17 +08:00
用 aria2 测试也是这样, 就算是 aria2 和 curl 混用也不能超过 2 个任务, 不知道为什么
daokedao
2022-06-11 19:11:43 +08:00
@Kisesy 我用 curl 和 chrome 混用也不能超过两个
AnroZ
2022-06-12 01:54:55 +08:00
刚刚测试了下,的确如此,第三个速度刚开始有速度但马上变为 0 了。
这应该 go http 包实现的问题,有空的话,可以仔细看下实现。
因为我换成 echo 框架,实现也是调用 http.ServeFile 的,测试到 5 个,都没问题。
haoliang
2022-06-12 11:41:07 +08:00
我觉得能确定的是:即使是单核 go 也能并发处理请求,GOMAXPROCS 并没有带来实质变化。
haoliang
2022-06-12 11:54:04 +08:00
读了一圈源码,并没有发现问题,敢问楼主啥硬件?
```
for { rw = l.accept(); go Conn(rw).serve(); }
```
不过 fileServer 没有用 sendfile 而是用 io.copy 让我比较惊讶
daokedao
2022-06-12 12:03:18 +08:00
@haoliang 我这台电脑 cpu 是 5700g ,系统是 Windows 11 ,Go 版本是 1.18.1 windows/amd64
Kisesy
2022-06-12 12:58:42 +08:00


感觉跟用 netutil 的 LimitListener 函数限制了并发一样
不一样的是如果用 LimitListener 限制了,这个 c 任务是读 0 字节,但这个会读 512 字节
AnroZ
2022-06-12 23:54:12 +08:00
而且,同样的代码在 WSL 子系统上运行是不存在这个问题
keepeye
2022-06-13 10:02:39 +08:00
我还是无法复现,curl 也改成 win 下的了,windows terminal + pwsh7 + curl ,4 个窗口同时下载,速度都正常

$ curl --limit-rate 1k http://127.0.0.1:8989/DesktopOK_Installer_x64.zip --output NUL
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
3 594k 3 18944 0 0 1021 0 0:09:56 0:00:18 0:09:38 1025


$ curl --limit-rate 1k http://127.0.0.1:8989/DesktopOK_Installer_x64.zip --output NUL
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
3 594k 3 18944 0 0 1021 0 0:09:56 0:00:18 0:09:38 1023


$ curl --limit-rate 1k http://127.0.0.1:8989/DesktopOK_Installer_x64.zip --output NUL
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
2 594k 2 15872 0 0 1020 0 0:09:57 0:00:15 0:09:42 1020


$ curl --limit-rate 1k http://127.0.0.1:8989/DesktopOK_Installer_x64.zip --output NUL
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
2 594k 2 15872 0 0 1020 0 0:09:57 0:00:15 0:09:42 1020
daokedao
2022-06-13 10:35:11 +08:00
我现在是按照 @AnroZ 的方法,换成 echo 或 gin ,测试都没有问题
whoami9894
2022-06-13 16:12:44 +08:00
http.FileServer 在 Windows 的最底层调用 TransmitFile ( https://github.com/golang/go/blob/master/src/internal/poll/sendfile_windows.go#L61)

TransmitFile 在个人机限制并发数为 2 ,服务器无限制 ( https://docs.microsoft.com/en-us/windows/win32/api/mswsock/nf-mswsock-transmitfile#remarks)

> Workstation and client versions of Windows optimize the TransmitFile function for minimum memory and resource utilization by limiting the number of concurrent TransmitFile operations allowed on the system to a maximum of two. On Windows Vista, Windows XP, Windows 2000 Professional, and Windows NT Workstation 3.51 and later only two outstanding TransmitFile requests are handled simultaneously; the third request will wait until one of the previous requests is completed.

Server versions of Windows optimize the TransmitFile function for high performance. On server versions, there are no default limits placed on the number of concurrent TransmitFile operations allowed on the system. Expect better performance results when using TransmitFile on server versions of Windows. On server versions of Windows, it is possible to set a limit on the maximum number of concurrent TransmitFile operations by creating a registry entry and setting a value for the following REG_DWORD:

HKEY_LOCAL_MACHINE\CurrentControlSet\Services\AFD\Parameters\MaxActiveTransmitFileCount
whoami9894
2022-06-13 16:49:59 +08:00
至于为什么 gin 可以,因为它根本没做这个优化

16 年有人发过 pr ( https://github.com/gin-gonic/gin/pull/638),但不知道什么原因关闭了,一直到今天都没实现这个优化



```go
// gin
8 0x000000000054620e in net.(*TCPConn).Write
at <autogenerated>:1
9 0x00000000005de533 in net/http.checkConnErrorWriter.Write
at d:/go/src/net/http/server.go:3532
10 0x0000000000592e75 in bufio.(*Writer).Write
at d:/go/src/bufio/bufio.go:639
11 0x00000000005d2695 in net/http.(*chunkWriter).Write
at d:/go/src/net/http/server.go:383
12 0x0000000000592e75 in bufio.(*Writer).Write
at d:/go/src/bufio/bufio.go:639
13 0x00000000005d819e in net/http.(*response).write
at d:/go/src/net/http/server.go:1592
14 0x00000000005d7ed0 in net/http.(*response).Write
at d:/go/src/net/http/server.go:1550
15 0x00000000006f3078 in github.com/gin-gonic/gin.(*responseWriter).Write
at c:/users/eddisonwang/go/pkg/mod/github.com/gin-gonic/gin@v1.8.1/response_writer.go:78
16 0x0000000000452444 in io.copyBuffer
at d:/go/src/io/io.go:425
17 0x00000000004520fa in io.Copy
at d:/go/src/io/io.go:382
18 0x00000000004520fa in io.CopyN
at d:/go/src/io/io.go:358
19 0x00000000005b2ba5 in net/http.serveContent
at d:/go/src/net/http/fs.go:337
20 0x00000000005b4af5 in net/http.serveFile
at d:/go/src/net/http/fs.go:664




// std http
6 0x0000000000f0ed5c in net.sendFile
at d:/go/src/net/sendfile_windows.go:37
7 0x0000000000f1196e in net.(*TCPConn).readFrom
at d:/go/src/net/tcpsock_posix.go:52
8 0x0000000000f10f76 in net.(*TCPConn).ReadFrom
at d:/go/src/net/tcpsock.go:104
9 0x0000000000fa08d9 in net/http.(*response).ReadFrom
at d:/go/src/net/http/server.go:597
10 0x0000000000e65a8b in io.copyBuffer
at d:/go/src/io/io.go:409
11 0x0000000000e657fa in io.Copy
at d:/go/src/io/io.go:382
12 0x0000000000e657fa in io.CopyN
at d:/go/src/io/io.go:358
13 0x0000000000f80465 in net/http.serveContent
at d:/go/src/net/http/fs.go:337
14 0x0000000000f823b5 in net/http.serveFile
at d:/go/src/net/http/fs.go:664
```
daokedao
2022-06-13 17:18:13 +08:00
@whoami9894 应该就是这个原因了,佩服 👍👍👍
Kisesy
2022-06-13 18:15:27 +08:00
@whoami9894 感谢啊,感觉就是有把大锁锁住了文件,原来是这个函数,微软也太抠了,限制到 2

这是一个专为移动设备优化的页面(即为了让你能够在 Google 搜索结果里秒开这个页面),如果你希望参与 V2EX 社区的讨论,你可以继续到 V2EX 上打开本讨论主题的完整版本。

https://tanronggui.xyz/t/858651

V2EX 是创意工作者们的社区,是一个分享自己正在做的有趣事物、交流想法,可以遇见新朋友甚至新机会的地方。

V2EX is a community of developers, designers and creative people.

© 2021 V2EX