24 Jun 2024

A stupid debugging experience

1. What happened

Servers SA and SB have the same docker installation, and the same running container CA and CB.
A Go file G can be built on CA, but on CB it reports this error:

runtime: failed to create new OS thread (have 2 already; errno=11)
runtime: may need to increase max user processes (ulimit -u)
fatal error: newosproc

I compared any related configurations between SA and SB. and between CA and CB, e.g., ulimit -a, /etc/security/limits.conf. They all look the same.
I created a new container CN on SA with the same docker image, CN can compile G.
I looked into the (complex) docker run script for CA/CB and figured out it was due to a resource constraint --pids-limit 100.
- Increasing this limit to 200 seems resolve the issue, but I had no idea why the Go compiler needed so many resources (perhaps due to package I imported).
Until this point, I realized, since the container did not support the compilation, why not just only transfer the compiled binary!
- How silly that I didn’t even try this in the beginning!
Since the program imports the net package, and there is a known issue of Alpine image running a Go binary file, I followed the post and disabled CGO on SA, then docker cp the binary to CA, and it worked.

The other day, I also spent hours debugging a route unreachable error when I want to send a request from CA to SA.
The CA is using the bridge network, so it should talk to SA via SA’s interface docker0 within the subnet 172.17.0.0/16.
However, in my case, the docker by default rejects packages from any container as shown in SA’s tcpdump result:

172.17.0.1->172.17.0.3 ICMP host unreachable- admin prohibited, length 68
By checking SA’s iptables, I found this rule:
```
  -A INPUT -j REJECT --reject-with icmp-host-prohibited
```
- Strangely, the ping still works with this rule.

In the end, I need to append a new rule to make the RPC work.

  iptables -I INPUT 1 -i docker0 -p tcp --dport <port> -s 172.17.0.0/16 -j ACCEPT