mirror of
https://gitee.com/xmz7891/ykxmz.git
synced 2026-10-18 13:40:45 +00:00
修开机慢 2 分钟：掉电重启后要 2 分 10 秒才能恢复
用户早上撞上 Cloudflare Error 1033。查出来是设备**掉电重启**：
上次开机到重启之间没有任何日志落盘（直接断电的特征），
dmesg 里还有 "l13: voltage operation not allowed"。
重启本身不是软件问题，但**恢复太慢**：
Startup finished in 3.846s (kernel) + 2min 6.358s (userspace) = 2分10秒
那 2 分钟里浏览器看到的就是 1033。
根因：
这台机器 NetworkManager 和 systemd-networkd 同时在跑，
但 wlan0 归 NetworkManager 管。
NetworkManager-wait-online -> 开机 9 秒成功
systemd-networkd-wait-online -> 等一个不归它管的网络，
干等 2 分钟超时，然后 failed
network-online.target 要等**两个都完成**，
被那个失败的拖了整整 2 分钟，
relay / tunnel / net-tune 全跟着等。
修法：
1. mask 掉 systemd-networkd-wait-online（网络不归它管，只会拖后腿）
2. watchdog 首次检查从"开机后 2 分钟"提前到 45 秒
实测（重启验证过）：
修之前 Startup finished in 3.846s + 2min 6.358s = 2分10秒
修之后 Startup finished in 3.218s + 14.060s = 17.3秒
掉电重启后公网恢复时间：**24 秒**（快了 5 倍多）
配套加了 scripts/speed-up-boot.sh，换设备时直接跑一遍。
顺带确认重启后一切自动恢复：
三个服务 active + enabled ✓
keepalive 30/10/3、MTU 探测、MSS 钳制、slow_start=0 全部持久化 ✓
隧道 4 条连接 http2 ✓
官网 / 管理页 / 截图全部 200 ✓
This commit is contained in:
xmz7891
@@ -1,23 +0,0 @@
|
||||
看门狗：连接数掉到 1 就立刻重启，不等攒够 3 次
|
||||
|
||||
排查公网间歇 530 时发现的：
|
||||
watchdog 日志显示「公网不通（HTTP 530，ha_connections=1）」
|
||||
—— 出问题那段时间隧道**只剩 1 条连接**。
|
||||
|
||||
原来 watchdog 只把 CONNS == 0 算"零连接"，
|
||||
CONNS == 1 不当回事。但实测**只剩一条连接时跨机房回源就会失败**
|
||||
（请求落 SEA，隧道只剩 LAX 一条）。
|
||||
|
||||
改法：
|
||||
* CONNS <= 1 都计入降级计数
|
||||
* 公网不通 **且** 连接数已在危险区 = 确认的故障（不是短抖动）
|
||||
-> 立刻重启，不等 3 次
|
||||
|
||||
每次检查间隔 20 秒，等 3 次就是白断 40 秒。现在能省下来。
|
||||
|
||||
实测：改完公网 15/15 = 100%，连接数 4 条。
|
||||
|
||||
顺带说明 watchdog 是怎么跑的：
|
||||
systemd timer（lanremote-watchdog.timer），每 20 秒调一次
|
||||
watchdog.service。之前查"服务是 inactive"一度以为它没在跑，
|
||||
其实是 timer 触发型的 —— oneshot 服务跑完就是 inactive，正常。
|
||||
@@ -0,0 +1,49 @@
|
||||
#!/bin/bash
|
||||
# 测"开机 -> 服务可用"要多久，看能不能缩短
|
||||
set -u
|
||||
|
||||
echo "════════ 1. 系统启动耗时 ════════"
|
||||
systemd-analyze 2>/dev/null | sed 's/^/ /'
|
||||
echo
|
||||
echo " 最慢的几项:"
|
||||
systemd-analyze blame 2>/dev/null | head -12 | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ 2. 关键服务的启动时刻 ════════"
|
||||
BOOT=$(date -d "$(uptime -s)" +%s 2>/dev/null || echo 0)
|
||||
for svc in lanremote-nettune lanremote-relay lanremote-tunnel; do
|
||||
ts=$(systemctl show "$svc" -p ActiveEnterTimestampMonotonic --value 2>/dev/null)
|
||||
if [ -n "$ts" ] && [ "$ts" != "0" ]; then
|
||||
sec=$((ts / 1000000))
|
||||
printf " %-24s 开机后 %s 秒\n" "$svc" "$sec"
|
||||
else
|
||||
printf " %-24s （读不到）\n" "$svc"
|
||||
fi
|
||||
done
|
||||
|
||||
echo
|
||||
echo "════════ 3. 隧道是什么时候连上的 ════════"
|
||||
grep -m3 'Registered tunnel connection' /root/lanremote-relay/tunnel.log 2>/dev/null \
|
||||
| sed 's/^/ /'
|
||||
echo " 当前时间: $(date -u '+%Y-%m-%dT%H:%M:%SZ')"
|
||||
echo " 日志里第一条注册时间戳对比一下就知道隔了多久"
|
||||
|
||||
echo
|
||||
echo "════════ 4. 开机第一条日志到隧道可用，中间卡在哪 ════════"
|
||||
journalctl -b --no-pager 2>/dev/null \
|
||||
| grep -E 'lanremote|cloudflared|Reached target network|Startup finished' \
|
||||
| head -20 | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ 5. network-online.target 花了多久 ════════"
|
||||
journalctl -b --no-pager -u NetworkManager -u systemd-networkd 2>/dev/null \
|
||||
| grep -iE 'online|carrier|dhcp|connected' | head -8 | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ 6. 有没有办法提前（当前依赖链）════════"
|
||||
systemctl show lanremote-tunnel -p After -p Wants -p Requires --value 2>/dev/null | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ 7. watchdog 间隔现在是多久 ════════"
|
||||
grep -E 'OnUnitActiveSec|OnBootSec' /etc/systemd/system/lanremote-watchdog.timer 2>/dev/null | sed 's/^/ /'
|
||||
systemctl cat lanremote-watchdog.timer 2>/dev/null | grep -E 'OnUnit|OnBoot' | sed 's/^/ /'
|
||||
@@ -0,0 +1,57 @@
|
||||
#!/bin/bash
|
||||
# 设备刚重启过（up 4 分钟）—— 查开机后各服务有没有自己起来
|
||||
set -u
|
||||
|
||||
echo "════════ 1. 开机多久了 ════════"
|
||||
uptime | sed 's/^/ /'
|
||||
echo " 开机于: $(who -b 2>/dev/null | awk '{print $3, $4}' || echo '?')"
|
||||
|
||||
echo
|
||||
echo "════════ 2. 三个服务 ════════"
|
||||
for svc in lanremote-relay lanremote-tunnel lanremote-nettune lanremote-watchdog.timer; do
|
||||
printf " %-26s %s (%s)\n" "$svc" "$(systemctl is-active $svc 2>&1)" "$(systemctl is-enabled $svc 2>&1)"
|
||||
done
|
||||
|
||||
echo
|
||||
echo "════════ 3. 开机自启清单里有没有它们 ════════"
|
||||
systemctl list-unit-files 2>/dev/null | grep -i lanremote | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ 4. 隧道连接数 ════════"
|
||||
echo " $(curl -s --max-time 5 http://127.0.0.1:20241/metrics 2>/dev/null | grep '^cloudflared_tunnel_ha_connections' | awk '{print $2}') 条"
|
||||
|
||||
echo
|
||||
echo "════════ 5. 网络调优参数还在吗 ════════"
|
||||
for k in tcp_keepalive_time tcp_keepalive_intvl tcp_mtu_probing tcp_slow_start_after_idle; do
|
||||
printf " %-30s = %s\n" "$k" "$(sysctl -n net.ipv4.$k 2>/dev/null || echo '?')"
|
||||
done
|
||||
echo " MSS 钳制规则: $(iptables -t mangle -S 2>/dev/null | grep -c 'TCPMSS' || echo 0) 条"
|
||||
|
||||
echo
|
||||
echo "════════ 6. 本机中转 ════════"
|
||||
echo " HTTP $(curl -s -o /dev/null -w '%{http_code}' --max-time 8 http://127.0.0.1:8080/v1/health)"
|
||||
|
||||
echo
|
||||
echo "════════ 7. 公网（浏览器实际走的）════════"
|
||||
echo " 隧道地址: $(cat /root/lanremote-relay/public_url.txt 2>/dev/null || echo '（没有）')"
|
||||
ok=0
|
||||
for i in $(seq 1 10); do
|
||||
c=$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 \
|
||||
-H 'User-Agent: LanRemote/1.4.0' https://relay.xxmxxs.top/v1/health)
|
||||
[ "$c" = "200" ] && ok=$((ok+1))
|
||||
printf " %2d: %s\n" "$i" "$c"
|
||||
sleep 1
|
||||
done
|
||||
echo " 可用 $ok/10"
|
||||
|
||||
echo
|
||||
echo "════════ 8. 新地址写进状态文件了吗 ════════"
|
||||
cat /root/lanremote-relay/public_url.txt 2>/dev/null | sed 's/^/ /'
|
||||
echo
|
||||
|
||||
echo "════════ 9. 当前 IP（换网络后）════════"
|
||||
hostname -I | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ 10. watchdog 最近记录 ════════"
|
||||
tail -8 /root/lanremote-relay/watchdog.log 2>/dev/null | sed 's/^/ /'
|
||||
@@ -0,0 +1,60 @@
|
||||
#!/bin/bash
|
||||
# 502/530 但隧道连着 —— 查回源配置（这是我唯一还没验证过的一环）
|
||||
set -u
|
||||
|
||||
BASE=http://127.0.0.1:8080
|
||||
|
||||
echo "════════ 1. 隧道 ingress 配置全文 ════════"
|
||||
cat /root/.cloudflared/config.yml 2>/dev/null | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ 2. relay 到底监听哪个地址 ════════"
|
||||
ss -lntp 2>/dev/null | grep 8080 | sed 's/^/ /'
|
||||
echo " ---"
|
||||
printf " 127.0.0.1:8080 -> HTTP %s\n" "$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 http://127.0.0.1:8080/v1/health)"
|
||||
printf " localhost:8080 -> HTTP %s\n" "$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 http://localhost:8080/v1/health)"
|
||||
printf " 192.168.10.105:8080 -> HTTP %s\n" "$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 http://192.168.10.105:8080/v1/health)"
|
||||
|
||||
echo
|
||||
echo "════════ 3. cloudflared 能看到 relay 吗（同一台机器）════════"
|
||||
echo " cloudflared 进程:"
|
||||
pgrep -af 'cloudflared tunnel' | head -2 | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ 4. 隧道 metrics 全量 ════════"
|
||||
curl -s --max-time 5 http://127.0.0.1:20241/metrics 2>/dev/null \
|
||||
| grep -E '^cloudflared_tunnel' | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ 5. 最近的注册/掉线（看 4 条是不是刚连上的）════════"
|
||||
tail -300 /root/lanremote-relay/tunnel.log 2>/dev/null \
|
||||
| grep -E 'Registered|Lost|Retrying|i/o timeout|error' \
|
||||
| tail -20 | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ 6. 公网连打 15 次（看是持续挂还是间歇）════════"
|
||||
ok=0
|
||||
for i in $(seq 1 15); do
|
||||
c=$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 \
|
||||
-H 'User-Agent: LanRemote/1.4.0' https://relay.xxmxxs.top/v1/health)
|
||||
[ "$c" = "200" ] && ok=$((ok+1))
|
||||
printf " %2d: %s\n" "$i" "$c"
|
||||
sleep 1
|
||||
done
|
||||
echo " 可用 $ok/15"
|
||||
|
||||
echo
|
||||
echo "════════ 7. 每次请求是不是落到不同机房 ════════"
|
||||
for i in 1 2 3 4 5 6; do
|
||||
ray=$(curl -sI --max-time 12 -H 'User-Agent: LanRemote/1.4.0' \
|
||||
https://relay.xxmxxs.top/v1/health 2>/dev/null \
|
||||
| grep -i '^cf-ray' | tr -d '\r' | awk '{print $2}')
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 12 \
|
||||
-H 'User-Agent: LanRemote/1.4.0' https://relay.xxmxxs.top/v1/health)
|
||||
printf " HTTP %s cf-ray %s\n" "$code" "$ray"
|
||||
done
|
||||
|
||||
echo
|
||||
echo "════════ 8. DNS 解析到哪 ════════"
|
||||
getent ahostsv4 relay.xxmxxs.top 2>/dev/null | awk '{print $1}' | sort -u | sed 's/^/ v4: /'
|
||||
getent ahostsv6 relay.xxmxxs.top 2>/dev/null | awk '{print $1}' | sort -u | head -3 | sed 's/^/ v6: /'
|
||||
@@ -0,0 +1,161 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
r'''把开机加速的修复固化到仓库（脚本 + 文档 + 提交）。
|
||||
|
||||
事故：用户早上撞上 Error 1033。
|
||||
查出设备**掉电重启**过（上次开机日志是空的 = 直接断电）。
|
||||
重启后要 **2 分 10 秒**公网才恢复 —— 那段时间浏览器就是 1033。
|
||||
|
||||
根因：
|
||||
这台机器上 NetworkManager 和 systemd-networkd 同时在跑，
|
||||
但 wlan0 归 **NetworkManager** 管。
|
||||
NetworkManager-wait-online -> 开机 9 秒就成功
|
||||
systemd-networkd-wait-online -> 等一个不归它管的网络，
|
||||
干等 2 分钟超时，然后 failed
|
||||
network-online.target 要等**两个都完成**，
|
||||
于是被那个失败的服务拖了整整 2 分钟，
|
||||
relay / tunnel / net-tune 全跟着等。
|
||||
|
||||
修法：
|
||||
1. mask 掉 systemd-networkd-wait-online（网络不归它管）
|
||||
2. watchdog 首次检查从"开机后 2 分钟"提前到 45 秒
|
||||
|
||||
实测：
|
||||
修之前 Startup finished in 3.846s + 2min 6.358s = 2分10秒
|
||||
修之后 Startup finished in 3.218s + 14.060s = 17.3秒
|
||||
掉电重启后公网恢复时间：**24 秒**
|
||||
'''
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(r"C:\Users\shu\Desktop\deepseekagent\LanRemote")
|
||||
GIT = r"C:\Program Files\Git\bin\git.exe"
|
||||
|
||||
# 把设备上的实际做法固化成脚本
|
||||
SCRIPT = '''#!/bin/bash
|
||||
# 加快掉电重启后的恢复速度。
|
||||
#
|
||||
# 背景（实测数据）：
|
||||
# 用户撞上的 Cloudflare Error 1033 是设备**掉电重启**造成的。
|
||||
# 修之前开机要 2 分 10 秒公网才恢复，那段时间浏览器就是 1033。
|
||||
#
|
||||
# 根因：这台机器 NetworkManager 和 systemd-networkd 同时在跑，
|
||||
# 但 wlan0 归 NetworkManager 管。
|
||||
# NetworkManager-wait-online -> 9 秒成功
|
||||
# systemd-networkd-wait-online -> 等不归它管的网络，
|
||||
# 干等 2 分钟超时然后 failed
|
||||
# network-online.target 要等两个都完成，
|
||||
# 被那个失败的拖了整整 2 分钟，
|
||||
# relay / tunnel / net-tune 全跟着等。
|
||||
#
|
||||
# 修完：用户空间 2分6秒 -> 14秒；掉电重启后 24 秒公网恢复。
|
||||
set -u
|
||||
|
||||
echo " 1. mask 掉 systemd-networkd-wait-online（网络不归它管）"
|
||||
systemctl disable --now systemd-networkd-wait-online.service 2>&1 | sed 's/^/ /'
|
||||
systemctl mask systemd-networkd-wait-online.service 2>&1 | sed 's/^/ /'
|
||||
|
||||
echo " 2. watchdog 首次检查提前到开机后 45 秒"
|
||||
cat > /etc/systemd/system/lanremote-watchdog.timer <<'EOF'
|
||||
[Unit]
|
||||
Description=每 20 秒检查一次隧道连接数
|
||||
|
||||
[Timer]
|
||||
# 原来 OnBootSec=2min。开机后隧道几十秒就连上了，
|
||||
# 干等 2 分钟才第一次检查没必要。
|
||||
OnBootSec=45s
|
||||
OnUnitActiveSec=20s
|
||||
AccuracySec=5s
|
||||
Unit=lanremote-watchdog.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
EOF
|
||||
systemctl daemon-reload
|
||||
systemctl restart lanremote-watchdog.timer
|
||||
|
||||
echo " 3. 确认"
|
||||
echo " networkd-wait-online: $(systemctl is-enabled systemd-networkd-wait-online 2>&1)（应为 masked）"
|
||||
echo " watchdog timer: $(systemctl is-active lanremote-watchdog.timer 2>&1)"
|
||||
systemctl list-dependencies network-online.target --no-pager 2>/dev/null | sed 's/^/ /'
|
||||
'''
|
||||
|
||||
MSG = """修开机慢 2 分钟：掉电重启后要 2 分 10 秒才能恢复
|
||||
|
||||
用户早上撞上 Cloudflare Error 1033。查出来是设备**掉电重启**：
|
||||
上次开机到重启之间没有任何日志落盘（直接断电的特征），
|
||||
dmesg 里还有 "l13: voltage operation not allowed"。
|
||||
|
||||
重启本身不是软件问题，但**恢复太慢**：
|
||||
Startup finished in 3.846s (kernel) + 2min 6.358s (userspace) = 2分10秒
|
||||
那 2 分钟里浏览器看到的就是 1033。
|
||||
|
||||
根因：
|
||||
这台机器 NetworkManager 和 systemd-networkd 同时在跑，
|
||||
但 wlan0 归 NetworkManager 管。
|
||||
NetworkManager-wait-online -> 开机 9 秒成功
|
||||
systemd-networkd-wait-online -> 等一个不归它管的网络，
|
||||
干等 2 分钟超时，然后 failed
|
||||
network-online.target 要等**两个都完成**，
|
||||
被那个失败的拖了整整 2 分钟，
|
||||
relay / tunnel / net-tune 全跟着等。
|
||||
|
||||
修法：
|
||||
1. mask 掉 systemd-networkd-wait-online（网络不归它管，只会拖后腿）
|
||||
2. watchdog 首次检查从"开机后 2 分钟"提前到 45 秒
|
||||
|
||||
实测（重启验证过）：
|
||||
修之前 Startup finished in 3.846s + 2min 6.358s = 2分10秒
|
||||
修之后 Startup finished in 3.218s + 14.060s = 17.3秒
|
||||
掉电重启后公网恢复时间：**24 秒**（快了 5 倍多）
|
||||
|
||||
配套加了 scripts/speed-up-boot.sh，换设备时直接跑一遍。
|
||||
|
||||
顺带确认重启后一切自动恢复：
|
||||
三个服务 active + enabled ✓
|
||||
keepalive 30/10/3、MTU 探测、MSS 钳制、slow_start=0 全部持久化 ✓
|
||||
隧道 4 条连接 http2 ✓
|
||||
官网 / 管理页 / 截图全部 200 ✓
|
||||
"""
|
||||
|
||||
|
||||
def save_script() -> None:
|
||||
target = ROOT / "relay" / "speed-up-boot.sh"
|
||||
target.write_text(SCRIPT, encoding="utf-8", newline="\n")
|
||||
print(" 已写 %s（%d 行）" % (target.name, SCRIPT.count("\n")))
|
||||
|
||||
|
||||
def commit() -> None:
|
||||
print()
|
||||
subprocess.run([GIT, "add", "-A"], cwd=str(ROOT), check=False)
|
||||
msg_file = ROOT / "_build" / "_c11.txt"
|
||||
msg_file.write_bytes(MSG.encode("utf-8"))
|
||||
r = subprocess.run([GIT, "commit", "-q", "-F", str(msg_file)],
|
||||
cwd=str(ROOT), capture_output=True, text=True,
|
||||
encoding="utf-8", errors="replace")
|
||||
msg_file.unlink(missing_ok=True)
|
||||
print(" commit exit=%s" % r.returncode)
|
||||
show = subprocess.run([GIT, "log", "--oneline", "-1"], cwd=str(ROOT),
|
||||
capture_output=True, text=True, encoding="utf-8",
|
||||
errors="replace")
|
||||
print(" " + (show.stdout or "").strip())
|
||||
env = dict(os.environ)
|
||||
env["GIT_SSH_COMMAND"] = r'"C:\Program Files\Git\usr\bin\ssh.exe"'
|
||||
push = subprocess.run([GIT, "push", "origin", "main"], cwd=str(ROOT),
|
||||
capture_output=True, text=True, encoding="utf-8",
|
||||
errors="replace", env=env)
|
||||
for line in (push.stderr or push.stdout or "").strip().splitlines()[-2:]:
|
||||
print(" " + line)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
save_script()
|
||||
commit()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+14
-2
@@ -13,9 +13,21 @@ from pathlib import Path
|
||||
|
||||
import paramiko
|
||||
|
||||
HOST = "192.168.2.156"
|
||||
# 随身 WiFi（Debian）的地址。
|
||||
#
|
||||
# 这台设备跟着教室网络走，地址会变：
|
||||
# 192.168.2.156 -> 192.168.10.105（换网络了）
|
||||
# 所以优先读环境变量 YKXMZ_DEV_HOST，没有才用下面的默认值。
|
||||
# 换网络时不用改代码：
|
||||
# $env:YKXMZ_DEV_HOST = "192.168.10.105"
|
||||
import os
|
||||
|
||||
HOST = os.environ.get("YKXMZ_DEV_HOST", "192.168.10.105")
|
||||
USER = "root"
|
||||
PASSWORD = "1313144"
|
||||
PASSWORD = os.environ.get("YKXMZ_DEV_PASS", "1313144")
|
||||
|
||||
# 旧地址留一份，方便回退或同时探测
|
||||
HOST_FALLBACKS = ("192.168.10.105", "192.168.2.156")
|
||||
|
||||
|
||||
def run(command: str, timeout: int = 600, quiet: bool = False) -> tuple:
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
#!/bin/bash
|
||||
# 修开机慢 2 分钟的根因
|
||||
#
|
||||
# 现象：
|
||||
# systemd-analyze 说用户空间花了 2 分 6 秒。
|
||||
# 日志显示 network-online.target 直到开机后 2 分 10 秒才达成，
|
||||
# 而 relay / tunnel / net-tune 全都在等它。
|
||||
#
|
||||
# 根因：
|
||||
# 这台机器上 NetworkManager 和 systemd-networkd **同时在跑**，
|
||||
# 但 wlan0 是 NetworkManager 管的。
|
||||
# NetworkManager-wait-online -> 开机 9 秒就成功
|
||||
# systemd-networkd-wait-online -> 等一个不归它管的网络，
|
||||
# 干等到 2 分钟超时，然后 **failed**
|
||||
# network-online.target 要等**两个都完成**，
|
||||
# 于是被那个失败的服务拖了整整 2 分钟。
|
||||
#
|
||||
# 修法：
|
||||
# 1. 屏蔽 systemd-networkd-wait-online（网络不归它管，它只会拖后腿）
|
||||
# 2. 顺手把 watchdog 首次检查从"开机后 2 分钟"提前到 45 秒
|
||||
# —— 开机后隧道本来就要几十秒才连上，早查早放心
|
||||
set -u
|
||||
|
||||
echo "════════ 1. 先确认网络确实归 NetworkManager 管 ════════"
|
||||
echo " wlan0 的持有者:"
|
||||
systemctl status NetworkManager --no-pager 2>/dev/null | head -3 | sed 's/^/ /'
|
||||
nmcli -t -f DEVICE,STATE,CONNECTION dev status 2>/dev/null | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ 2. 屏蔽 systemd-networkd-wait-online ════════"
|
||||
systemctl disable --now systemd-networkd-wait-online.service 2>&1 | sed 's/^/ /'
|
||||
systemctl mask systemd-networkd-wait-online.service 2>&1 | sed 's/^/ /'
|
||||
echo " 现在状态: $(systemctl is-enabled systemd-networkd-wait-online.service 2>&1) / $(systemctl is-active systemd-networkd-wait-online.service 2>&1)"
|
||||
|
||||
echo
|
||||
echo "════════ 3. 确认 network-online.target 不再依赖它 ════════"
|
||||
systemctl list-dependencies network-online.target --no-pager 2>/dev/null | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ 4. watchdog 首次检查提前到 45 秒 ════════"
|
||||
cat > /etc/systemd/system/lanremote-watchdog.timer <<'EOF'
|
||||
[Unit]
|
||||
Description=每 20 秒检查一次隧道连接数
|
||||
|
||||
[Timer]
|
||||
# 原来 OnBootSec=2min —— 开机后隧道几十秒就连上了，
|
||||
# 干等 2 分钟才第一次检查没必要。
|
||||
OnBootSec=45s
|
||||
OnUnitActiveSec=20s
|
||||
AccuracySec=5s
|
||||
Unit=lanremote-watchdog.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
EOF
|
||||
systemctl daemon-reload
|
||||
systemctl restart lanremote-watchdog.timer
|
||||
echo " $(systemctl is-active lanremote-watchdog.timer) / $(systemctl is-enabled lanremote-watchdog.timer)"
|
||||
|
||||
echo
|
||||
echo "════════ 5. 服务还正常吗 ════════"
|
||||
for svc in lanremote-relay lanremote-tunnel lanremote-nettune; do
|
||||
printf " %-22s %s\n" "$svc" "$(systemctl is-active $svc 2>&1)"
|
||||
done
|
||||
echo " 隧道连接数: $(curl -s --max-time 5 http://127.0.0.1:20241/metrics 2>/dev/null | grep '^cloudflared_tunnel_ha_connections' | awk '{print $2}')"
|
||||
echo " 本机 relay: HTTP $(curl -s -o /dev/null -w '%{http_code}' --max-time 8 http://127.0.0.1:8080/v1/health)"
|
||||
|
||||
echo
|
||||
echo "════════ 6. 公网还通吗 ════════"
|
||||
ok=0
|
||||
for i in $(seq 1 5); do
|
||||
c=$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 \
|
||||
-H 'User-Agent: LanRemote/1.4.0' https://relay.xxmxxs.top/v1/health)
|
||||
[ "$c" = "200" ] && ok=$((ok+1))
|
||||
printf " %d: %s\n" "$i" "$c"
|
||||
sleep 1
|
||||
done
|
||||
echo " 可用 $ok/5"
|
||||
@@ -0,0 +1,57 @@
|
||||
#!/bin/bash
|
||||
# 重启后全面体检
|
||||
set -u
|
||||
|
||||
echo "════════ 开机信息 ════════"
|
||||
uptime -p | sed 's/^/ /'
|
||||
systemd-analyze 2>/dev/null | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ 服务 ════════"
|
||||
for s in lanremote-relay lanremote-tunnel lanremote-nettune lanremote-watchdog.timer NetworkManager-wait-online; do
|
||||
printf " %-30s %-10s %s\n" "$s" "$(systemctl is-active $s 2>&1)" "$(systemctl is-enabled $s 2>&1)"
|
||||
done
|
||||
echo " ---"
|
||||
echo " systemd-networkd-wait-online: $(systemctl is-enabled systemd-networkd-wait-online 2>&1) （应该是 masked）"
|
||||
|
||||
echo
|
||||
echo "════════ 网络调优是否持久化 ════════"
|
||||
for k in tcp_keepalive_time tcp_keepalive_intvl tcp_keepalive_probes tcp_mtu_probing tcp_slow_start_after_idle tcp_congestion_control; do
|
||||
printf " %-32s = %s\n" "$k" "$(sysctl -n net.ipv4.$k 2>/dev/null || echo '?')"
|
||||
done
|
||||
echo " MSS 钳制: $(iptables -t mangle -S 2>/dev/null | grep -c TCPMSS) 条"
|
||||
|
||||
echo
|
||||
echo "════════ 隧道 ════════"
|
||||
echo " 连接数: $(curl -s --max-time 5 http://127.0.0.1:20241/metrics 2>/dev/null | grep '^cloudflared_tunnel_ha_connections' | awk '{print $2}')"
|
||||
echo " 落点:"
|
||||
grep -E 'Registered tunnel connection' /root/lanremote-relay/tunnel.log 2>/dev/null | tail -4 \
|
||||
| grep -oE 'location=[a-z0-9]+ protocol=[a-z0-9]+' | sed 's/^/ /'
|
||||
echo " 参数:"
|
||||
grep -oE '\-\-protocol [a-z0-9]+|--ha-connections [0-9]+' /etc/systemd/system/lanremote-tunnel.service | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ 本机中转 ════════"
|
||||
echo " HTTP $(curl -s -o /dev/null -w '%{http_code}' --max-time 8 http://127.0.0.1:8080/v1/health)"
|
||||
|
||||
echo
|
||||
echo "════════ 公网 12 次 ════════"
|
||||
ok=0
|
||||
for i in $(seq 1 12); do
|
||||
c=$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 \
|
||||
-H 'User-Agent: LanRemote/1.4.0' https://relay.xxmxxs.top/v1/health)
|
||||
[ "$c" = "200" ] && ok=$((ok+1))
|
||||
printf " %2d: %s\n" "$i" "$c"
|
||||
sleep 1
|
||||
done
|
||||
echo " 可用 $ok/12"
|
||||
|
||||
echo
|
||||
echo "════════ 官网 + 管理页 ════════"
|
||||
echo " 官网 HTTP $(curl -s -o /dev/null -w '%{http_code}' --max-time 15 https://relay.xxmxxs.top/)"
|
||||
echo " 管理页 HTTP $(curl -s -o /dev/null -w '%{http_code}' --max-time 15 https://relay.xxmxxs.top/zhongzhuanzhan)"
|
||||
echo " 截图 HTTP $(curl -s -o /dev/null -w '%{http_code}' --max-time 15 https://relay.xxmxxs.top/shot/pc-main.png)"
|
||||
|
||||
echo
|
||||
echo "════════ watchdog 日志 ════════"
|
||||
tail -6 /root/lanremote-relay/watchdog.log 2>/dev/null | sed 's/^/ /'
|
||||
@@ -0,0 +1,29 @@
|
||||
#!/bin/bash
|
||||
# 重启验证：开机到公网可用要多久
|
||||
#
|
||||
# 背景：用户撞上的 1033 是设备**掉电重启**造成的。
|
||||
# 修之前 systemd-analyze 显示用户空间花了 2 分 6 秒，
|
||||
# 因为 systemd-networkd-wait-online 等一个不归它管的网络，
|
||||
# 干等 2 分钟超时失败，把 network-online.target 一起拖住。
|
||||
# 已经把它 mask 掉了。
|
||||
#
|
||||
# 这个脚本：记录重启前的状态 -> 重启 -> 从外面轮询公网 ->
|
||||
# 记录它多久恢复。
|
||||
set -u
|
||||
|
||||
MARK=/root/reboot_test_mark.txt
|
||||
echo " 当前开机时长: $(uptime -p)"
|
||||
echo " 当前时间: $(date '+%H:%M:%S')"
|
||||
echo " 服务状态:"
|
||||
for svc in lanremote-relay lanremote-tunnel lanremote-nettune; do
|
||||
printf " %-22s %s\n" "$svc" "$(systemctl is-active $svc)"
|
||||
done
|
||||
echo " 隧道连接: $(curl -s --max-time 5 http://127.0.0.1:20241/metrics 2>/dev/null | grep '^cloudflared_tunnel_ha_connections' | awk '{print $2}')"
|
||||
echo " 公网: HTTP $(curl -s -o /dev/null -w '%{http_code}' --max-time 15 -H 'User-Agent: LanRemote/1.4.0' https://relay.xxmxxs.top/v1/health)"
|
||||
|
||||
date +%s > "$MARK"
|
||||
echo
|
||||
echo " 重启前的时间戳已写入 $MARK: $(cat $MARK)"
|
||||
echo " **现在重启** —— 大概 1 分钟内会断开"
|
||||
sleep 2
|
||||
reboot
|
||||
@@ -0,0 +1,29 @@
|
||||
#!/bin/bash
|
||||
# 看当前依赖链，准备去掉 network-online 的阻塞
|
||||
set -u
|
||||
|
||||
echo "════════ relay unit ════════"
|
||||
systemctl cat lanremote-relay 2>/dev/null | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ tunnel unit ════════"
|
||||
systemctl cat lanremote-tunnel 2>/dev/null | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ net-tune unit ════════"
|
||||
systemctl cat lanremote-nettune 2>/dev/null | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ watchdog timer ════════"
|
||||
systemctl cat lanremote-watchdog.timer 2>/dev/null | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ 为什么 network-online 这么久 ════════"
|
||||
echo " NetworkManager 等待配置的单元:"
|
||||
systemctl list-units --type=service --state=running 2>/dev/null | grep -iE 'network|wpa|dhcp' | sed 's/^/ /'
|
||||
echo
|
||||
echo " NetworkManager-wait-online 状态:"
|
||||
systemctl status NetworkManager-wait-online.service --no-pager 2>/dev/null | head -6 | sed 's/^/ /'
|
||||
echo
|
||||
echo " systemd-networkd-wait-online 状态:"
|
||||
systemctl status systemd-networkd-wait-online.service --no-pager 2>/dev/null | head -6 | sed 's/^/ /'
|
||||
@@ -0,0 +1,56 @@
|
||||
#!/bin/bash
|
||||
# 设备重启了 —— 查是掉电、还是崩溃、还是被谁重启的
|
||||
set -u
|
||||
|
||||
echo "════════ 1. 这次开机的起点 ════════"
|
||||
uptime | sed 's/^/ /'
|
||||
echo " 当前时间: $(date '+%m-%d %H:%M:%S')"
|
||||
echo " 开机时刻: $(uptime -s 2>/dev/null || echo '?')"
|
||||
|
||||
echo
|
||||
echo "════════ 2. 上一次关机是干净的还是掉电 ════════"
|
||||
echo " --- last reboot ---"
|
||||
last -x reboot 2>/dev/null | head -6 | sed 's/^/ /'
|
||||
echo " --- last shutdown ---"
|
||||
last -x shutdown 2>/dev/null | head -4 | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ 3. 本次启动的日志开头（看有没有异常）════════"
|
||||
journalctl -b --no-pager 2>/dev/null | head -25 | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ 4. 有没有内核崩溃 / 掉电迹象 ════════"
|
||||
echo " --- 上次启动末尾（也就是重启前最后的日志）---"
|
||||
journalctl -b -1 --no-pager 2>/dev/null | tail -25 | sed 's/^/ /'
|
||||
echo
|
||||
echo " （如果上面是空的，说明上次的日志没落盘 —— 典型的直接掉电）"
|
||||
|
||||
echo
|
||||
echo "════════ 5. dmesg 里的告警 ════════"
|
||||
dmesg 2>/dev/null | grep -iE 'error|warn|fail|thermal|voltage|under' | tail -12 | sed 's/^/ /'
|
||||
echo " （没输出=没有明显告警）"
|
||||
|
||||
echo
|
||||
echo "════════ 6. 温度 / 电压（如果是过热或欠压会重启）════════"
|
||||
for f in /sys/class/thermal/thermal_zone*/temp; do
|
||||
[ -f "$f" ] && echo " $f: $(($(cat $f) / 1000))°C"
|
||||
done
|
||||
command -v vcgencmd >/dev/null 2>&1 && vcgencmd measure_temp 2>/dev/null | sed 's/^/ /'
|
||||
cat /sys/class/power_supply/*/voltage_now 2>/dev/null | while read v; do
|
||||
echo " 电压: $((v / 1000)) mV"
|
||||
done
|
||||
|
||||
echo
|
||||
echo "════════ 7. 内存压力（OOM 会把系统搞挂）════════"
|
||||
free -m | sed 's/^/ /'
|
||||
echo " 本次启动有没有 OOM:"
|
||||
journalctl -b --no-pager 2>/dev/null | grep -ic 'out of memory\|oom-kill' | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ 8. 是不是被关机命令重启的 ════════"
|
||||
journalctl -b --no-pager 2>/dev/null | grep -iE 'shutdown|reboot|poweroff|systemd-logind' | head -8 | sed 's/^/ /'
|
||||
|
||||
echo
|
||||
echo "════════ 9. 日志持久化了吗（掉电的话日志会丢）════════"
|
||||
echo " /var/log/journal 存在: $(if [ -d /var/log/journal ]; then echo 是; else echo '否（日志只在内存里，重启就没）'; fi)"
|
||||
echo " 当前 journal 占用: $(journalctl --disk-usage 2>/dev/null | sed 's/.*take up //' || echo '?')"
|
||||
Reference in New Issue
Block a user