mirror of
https://gitee.com/xmz7891/ykxmz.git
synced 2026-10-18 12:30:44 +00:00
监测单元和预案重写成"分层"版(来源改为本地整份文件)
== 为什么重写 ==
certbot 签证书时加了 http -> https 的 301,而监测单元原来判断
"curl 拿到 200 才算健康"、且 curl **默认不跟跳转** -> 拿到 301
-> 判成"网站挂了" -> 又要启动预案 -> 重启中转 -> 再查还是 301。
这是**第二次**踩同一类坑了(上一次是改监听地址忘了改探测地址)。
共同教训:**改了入口的行为,必须回头看所有探测入口的地方。**
== 新设计:分两层探测 ==
PRIMARY https://yk.xxmxxs.top/v1/health 用户实际走的路
FALLBACK http://127.0.0.1:13245/v1/health 中转本身
两个都查才能分清哪一层坏了,进而选对抢救动作:
PRIMARY 挂 + FALLBACK 好 -> edge 层(CF / DNS / nginx)
两个都挂 -> core 层(中转服务本身)
预案据此走不同的阶梯:
edge: 重载 nginx -> 重启 nginx -> 查 443 与证书
core: 重启中转 -> 强杀重拉 -> 中转+nginx 都重启
不区分层次的话只能瞎重启,而 edge 层的问题重启中转一万次也没用。
== 做法上的改进:不再远端打补丁 ==
前几轮一直是"SSH 过去用嵌套 heredoc 现场改 monitor.sh",
连续踩三次:PowerShell 在本机求值 \、引号被吃、
脚本里的 %{http_code} 和 Python 的 %% 格式化打架。
现在改成:relay/monitor-lightsail.sh 和 relay/rescue-lightsail.sh
是**唯一事实来源**,本地编辑 -> 整份上传 -> 重装,
改了什么一目了然,也能进 git。
== 顺带 ==
因为 Lightsail 防火墙还没放行 443,PRIMARY 一直是 000,
监测会把 fails 一直加上去、白生成抢救报告。
所以暂时把 monitor.timer 停掉,等 443 开了再启用
(脚本 _build/pause_monitor.py,恢复是一条 systemctl start)。
This commit is contained in:
@@ -0,0 +1,44 @@
|
|||||||
|
监测单元和预案重写成"分层"版(来源改为本地整份文件)
|
||||||
|
|
||||||
|
== 为什么重写 ==
|
||||||
|
|
||||||
|
certbot 签证书时加了 http -> https 的 301,而监测单元原来判断
|
||||||
|
"curl 拿到 200 才算健康"、且 curl **默认不跟跳转** -> 拿到 301
|
||||||
|
-> 判成"网站挂了" -> 又要启动预案 -> 重启中转 -> 再查还是 301。
|
||||||
|
这是**第二次**踩同一类坑了(上一次是改监听地址忘了改探测地址)。
|
||||||
|
|
||||||
|
共同教训:**改了入口的行为,必须回头看所有探测入口的地方。**
|
||||||
|
|
||||||
|
== 新设计:分两层探测 ==
|
||||||
|
|
||||||
|
PRIMARY https://yk.xxmxxs.top/v1/health 用户实际走的路
|
||||||
|
FALLBACK http://127.0.0.1:13245/v1/health 中转本身
|
||||||
|
|
||||||
|
两个都查才能分清哪一层坏了,进而选对抢救动作:
|
||||||
|
|
||||||
|
PRIMARY 挂 + FALLBACK 好 -> edge 层(CF / DNS / nginx)
|
||||||
|
两个都挂 -> core 层(中转服务本身)
|
||||||
|
|
||||||
|
预案据此走不同的阶梯:
|
||||||
|
|
||||||
|
edge: 重载 nginx -> 重启 nginx -> 查 443 与证书
|
||||||
|
core: 重启中转 -> 强杀重拉 -> 中转+nginx 都重启
|
||||||
|
|
||||||
|
不区分层次的话只能瞎重启,而 edge 层的问题重启中转一万次也没用。
|
||||||
|
|
||||||
|
== 做法上的改进:不再远端打补丁 ==
|
||||||
|
|
||||||
|
前几轮一直是"SSH 过去用嵌套 heredoc 现场改 monitor.sh",
|
||||||
|
连续踩三次:PowerShell 在本机求值 \、引号被吃、
|
||||||
|
脚本里的 %{http_code} 和 Python 的 %% 格式化打架。
|
||||||
|
|
||||||
|
现在改成:relay/monitor-lightsail.sh 和 relay/rescue-lightsail.sh
|
||||||
|
是**唯一事实来源**,本地编辑 -> 整份上传 -> 重装,
|
||||||
|
改了什么一目了然,也能进 git。
|
||||||
|
|
||||||
|
== 顺带 ==
|
||||||
|
|
||||||
|
因为 Lightsail 防火墙还没放行 443,PRIMARY 一直是 000,
|
||||||
|
监测会把 fails 一直加上去、白生成抢救报告。
|
||||||
|
所以暂时把 monitor.timer 停掉,等 443 开了再启用
|
||||||
|
(脚本 _build/pause_monitor.py,恢复是一条 systemctl start)。
|
||||||
@@ -0,0 +1,171 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
r'''监测单元适配"80 会 301 到 https"之后的新情况。
|
||||||
|
|
||||||
|
== 问题 ==
|
||||||
|
|
||||||
|
certbot 签证书时加了 `--redirect`,于是 nginx 的 80 端口现在这样:
|
||||||
|
|
||||||
|
listen 80; server_name yk.xxmxxs.top;
|
||||||
|
return 301 https://$host$request_uri;
|
||||||
|
|
||||||
|
而监测单元原来的判断是"curl 拿到 200 才算健康"。curl **默认不跟跳转**,
|
||||||
|
于是它拿到 301 -> 判定失败 -> 连续 3 次就启动紧急预案
|
||||||
|
-> 预案重启中转服务 -> 再查还是 301 -> **又一个无限循环**。
|
||||||
|
|
||||||
|
这个坑和刚才那个"改监听地址忘了改探测地址"是同一类:
|
||||||
|
**改了入口的行为,必须回头看所有探测入口的地方。**
|
||||||
|
|
||||||
|
== 改法 ==
|
||||||
|
|
||||||
|
把探测地址和判断都改成"看用户真实路径":
|
||||||
|
|
||||||
|
PRIMARY https://yk.xxmxxs.top/v1/health (经 Cloudflare,用户实际走的)
|
||||||
|
FALLBACK http://127.0.0.1:13245/v1/health (中转本身,绕开 CF 和 nginx)
|
||||||
|
|
||||||
|
判断规则也放宽一点:
|
||||||
|
* curl 加 -L 跟跳转(301 本身不算故障)
|
||||||
|
* 200 才算健康
|
||||||
|
* **如果 PRIMARY 挂着但 FALLBACK 好**,说明问题在
|
||||||
|
Cloudflare / DNS / nginx 那一层 —— 这时需要的是
|
||||||
|
"重启 nginx",而不是重启中转。所以预案里补一级。
|
||||||
|
|
||||||
|
== 为什么 PRIMARY 用 https 而不是 http ==
|
||||||
|
|
||||||
|
因为 http 现在只是个跳转,真正的服务在 https 上。
|
||||||
|
用 https 探测,用户遇到什么我们就测到什么。
|
||||||
|
(443 端口还没在 Lightsail 防火墙开之前,PRIMARY 会失败、
|
||||||
|
FALLBACK 成功 —— 这时不会误判成"中转挂了",见上面的规则。)
|
||||||
|
'''
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
sys.path.insert(0, str(ROOT / "_build"))
|
||||||
|
from lssh import run # noqa: E402
|
||||||
|
|
||||||
|
RELAY_DIR = "/opt/ykxmz-relay"
|
||||||
|
|
||||||
|
NEW_HEAD = '''# ============================================================================
|
||||||
|
# 探测地址
|
||||||
|
# ============================================================================
|
||||||
|
#
|
||||||
|
# PRIMARY:用户实际走的路(经 Cloudflare -> nginx -> 中转,全程 https)
|
||||||
|
# FALLBACK:中转本身,绕开 Cloudflare 和 nginx
|
||||||
|
#
|
||||||
|
# ⚠ 两个都查是有意的:
|
||||||
|
# * PRIMARY 挂 + FALLBACK 好 -> 问题在 CF / DNS / nginx 层
|
||||||
|
# * 两个都挂 -> 中转服务本身出问题了
|
||||||
|
# 分开判才能选对抢救动作(重启 nginx 还是重启中转)。
|
||||||
|
PRIMARY_URL="${PRIMARY_URL:-https://yk.xxmxxs.top}"
|
||||||
|
FALLBACK_URL="${FALLBACK_URL:-http://127.0.0.1:13245}"
|
||||||
|
'''
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
print("=" * 62)
|
||||||
|
print(" 监测单元适配 https + 301")
|
||||||
|
print("=" * 62)
|
||||||
|
|
||||||
|
print()
|
||||||
|
print(" ── 1. 改探测地址那几行 ──")
|
||||||
|
script = (
|
||||||
|
"python3 - <<'PYEOF'\n"
|
||||||
|
"from pathlib import Path\n"
|
||||||
|
"p = Path('%s/monitor.sh')\n"
|
||||||
|
"lines = p.read_text(encoding='utf-8').splitlines()\n"
|
||||||
|
"out = []\n"
|
||||||
|
"done = False\n"
|
||||||
|
"for l in lines:\n"
|
||||||
|
" if l.startswith('PUBLIC_URL=') or l.startswith('FALLBACK_URL='):\n"
|
||||||
|
" continue\n"
|
||||||
|
" if l.strip() == '# 探测地址:域名优先(用户实际走的路),IP:80 兜底' and not done:\n"
|
||||||
|
" out.append(%r)\n"
|
||||||
|
" done = True\n"
|
||||||
|
" continue\n"
|
||||||
|
" out.append(l)\n"
|
||||||
|
"p.write_text('\\n'.join(out) + '\\n', encoding='utf-8', newline='\\n')\n"
|
||||||
|
"print(' 替换完成: %s' % done)\n"
|
||||||
|
"PYEOF" % (RELAY_DIR, NEW_HEAD))
|
||||||
|
out = run(script, timeout=120, quiet=True)
|
||||||
|
for ln in out[1].strip().splitlines():
|
||||||
|
print(" " + ln)
|
||||||
|
|
||||||
|
print()
|
||||||
|
print(" ── 2. 健康检查改成 -L + 分清两层 ──")
|
||||||
|
script2 = (
|
||||||
|
"python3 - <<'PYEOF'\n"
|
||||||
|
"from pathlib import Path\n"
|
||||||
|
"p = Path('%s/monitor.sh')\n"
|
||||||
|
"t = p.read_text(encoding='utf-8')\n"
|
||||||
|
"\n"
|
||||||
|
"old = '''PROBE=\"$PUBLIC_URL\"\n"
|
||||||
|
"CODE=$(curl -s -o /dev/null -w '%%{http_code}' --max-time \"$CHECK_TIMEOUT\" \"$PROBE/v1/health\" 2>/dev/null)\n"
|
||||||
|
"if [ \"$CODE\" != \"200\" ] && [ -n \"$FALLBACK_URL\" ]; then\n"
|
||||||
|
" PROBE=\"$FALLBACK_URL\"\n"
|
||||||
|
" CODE=$(curl -s -o /dev/null -w '%%{http_code}' --max-time \"$CHECK_TIMEOUT\" \"$PROBE/v1/health\" 2>/dev/null)\n"
|
||||||
|
"fi'''\n"
|
||||||
|
"\n"
|
||||||
|
"new = '''# -L 跟跳转:nginx 那边 http 会 301 到 https,\n"
|
||||||
|
"# 不跟跳转的话拿到 301 会被误判成故障(这坑踩过一次)。\n"
|
||||||
|
"PUB_CODE=$(curl -sL -o /dev/null -w '%%{http_code}' --max-time \"$CHECK_TIMEOUT\" \"$PRIMARY_URL/v1/health\" 2>/dev/null)\n"
|
||||||
|
"LOC_CODE=$(curl -s -o /dev/null -w '%%{http_code}' --max-time \"$CHECK_TIMEOUT\" \"$FALLBACK_URL/v1/health\" 2>/dev/null)\n"
|
||||||
|
"CODE=\"$PUB_CODE\"\n"
|
||||||
|
"LAYER=\"\"\n"
|
||||||
|
"if [ \"$PUB_CODE\" != \"200\" ] && [ \"$LOC_CODE\" = \"200\" ]; then\n"
|
||||||
|
" LAYER=\"edge\" # 中转是好的,问题在 CF / DNS / nginx 层\n"
|
||||||
|
"elif [ \"$PUB_CODE\" != \"200\" ] && [ \"$LOC_CODE\" != \"200\" ]; then\n"
|
||||||
|
" LAYER=\"core\" # 中转本身也不行\n"
|
||||||
|
"fi'''\n"
|
||||||
|
"\n"
|
||||||
|
"if old in t:\n"
|
||||||
|
" t = t.replace(old, new, 1)\n"
|
||||||
|
" print(' ok 健康检查替换完成')\n"
|
||||||
|
"else:\n"
|
||||||
|
" print(' x 没匹配上,实际内容:')\n"
|
||||||
|
" i = t.find('CODE=$(curl')\n"
|
||||||
|
" print(t[max(0,i-200):i+400])\n"
|
||||||
|
"p.write_text(t, encoding='utf-8', newline='\\n')\n"
|
||||||
|
"PYEOF" % RELAY_DIR)
|
||||||
|
out = run(script2, timeout=120, quiet=True)
|
||||||
|
for ln in out[1].strip().splitlines():
|
||||||
|
print(" " + ln)
|
||||||
|
|
||||||
|
print()
|
||||||
|
print(" ── 3. 语法检查 + 跑一次 ──")
|
||||||
|
out = run("bash -n %s/monitor.sh && echo ' 语法 OK'; "
|
||||||
|
"printf 'fails=0\\nlast_rescue=0\\n' > "
|
||||||
|
"%s/monitor_state.txt; "
|
||||||
|
"bash %s/monitor.sh; echo ' --- 状态 ---'; "
|
||||||
|
"cat %s/monitor_state.txt; echo ' --- 日志尾 ---'; "
|
||||||
|
"grep -v 'Permission denied' %s/monitor.log | tail -4"
|
||||||
|
% (RELAY_DIR, RELAY_DIR, RELAY_DIR, RELAY_DIR, RELAY_DIR),
|
||||||
|
timeout=180, quiet=True)
|
||||||
|
for ln in out[1].strip().splitlines():
|
||||||
|
print(" " + ln)
|
||||||
|
|
||||||
|
print()
|
||||||
|
print(" ── 4. 现在探测的各层 ──")
|
||||||
|
out = run(
|
||||||
|
'printf " %%-34s %%s\\n" "PRIMARY https://域名" '
|
||||||
|
'"$(curl -sL -o /dev/null -w \'%{http_code}\' --max-time 15 '
|
||||||
|
'https://yk.xxmxxs.top/v1/health)"; '
|
||||||
|
'printf " %%-34s %%s\\n" "FALLBACK 127.0.0.1:13245" '
|
||||||
|
'"$(curl -s -o /dev/null -w \'%{http_code}\' --max-time 8 '
|
||||||
|
'http://127.0.0.1:13245/v1/health)"; '
|
||||||
|
'printf " %%-34s %%s\\n" "nginx 本机:80(会 301)" '
|
||||||
|
'"$(curl -s -o /dev/null -w \'%{http_code}\' --max-time 8 '
|
||||||
|
'http://127.0.0.1:80/v1/health)"; '
|
||||||
|
'printf " %%-34s %%s\\n" "nginx 本机:443" '
|
||||||
|
'"$(curl -sk -o /dev/null -w \'%{http_code}\' --max-time 8 '
|
||||||
|
'--resolve yk.xxmxxs.top:443:127.0.0.1 '
|
||||||
|
'https://yk.xxmxxs.top/v1/health)"',
|
||||||
|
timeout=180, quiet=True)
|
||||||
|
for ln in out[1].strip().splitlines():
|
||||||
|
print(ln)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,112 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
r'''把本地的 monitor/rescue 脚本装到 Lightsail(第二版:https 版)。
|
||||||
|
|
||||||
|
== 为什么单独写一个,而不是继续远端打补丁 ==
|
||||||
|
|
||||||
|
前面几轮我一直在用"SSH 过去、用嵌套 heredoc 现场改 monitor.sh"的做法,
|
||||||
|
结果连续踩了三次:
|
||||||
|
|
||||||
|
1. PowerShell 的 `$(...)` 在**本机**被求值(报"systemctl 不是 cmdlet")
|
||||||
|
2. 引号被吃掉 -> Python 报 unterminated string literal
|
||||||
|
3. 脚本里的 `%{http_code}` 和 Python 的 `%` 格式化打架
|
||||||
|
-> "not enough arguments for format string"
|
||||||
|
|
||||||
|
根本问题:**远端打补丁这件事本身就脆弱**。
|
||||||
|
本地写完整文件、整份上传,就没有这些问题 —— 而且改了什么一目了然,
|
||||||
|
还能进 git 做版本管理。
|
||||||
|
|
||||||
|
所以现在改成:relay/monitor-lightsail.sh 和 relay/rescue-lightsail.sh
|
||||||
|
是**唯一事实来源**,本地编辑 -> 整份上传 -> 重装。
|
||||||
|
|
||||||
|
== 本地文件名 vs 远端文件名 ==
|
||||||
|
|
||||||
|
本地叫 monitor-lightsail.sh(和那台旧的 monitor.sh 区分开,
|
||||||
|
免得看混),远端统一叫 monitor.sh / rescue.sh,
|
||||||
|
因为 systemd unit 和监测脚本里写的是这个名字。
|
||||||
|
'''
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
sys.path.insert(0, str(ROOT / "_build"))
|
||||||
|
from lssh import run, upload # noqa: E402
|
||||||
|
|
||||||
|
RELAY_DIR = "/opt/ykxmz-relay"
|
||||||
|
|
||||||
|
# (本地文件, 远端文件名)
|
||||||
|
PAIRS = [
|
||||||
|
("relay/monitor-lightsail.sh", "monitor.sh"),
|
||||||
|
("relay/rescue-lightsail.sh", "rescue.sh"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
print("=" * 62)
|
||||||
|
print(" 装 monitor / rescue 到 Lightsail")
|
||||||
|
print("=" * 62)
|
||||||
|
|
||||||
|
print()
|
||||||
|
print(" ── 1. 上传 ──")
|
||||||
|
for local_name, remote_name in PAIRS:
|
||||||
|
local = ROOT / local_name
|
||||||
|
if not local.is_file():
|
||||||
|
print(" ✗ 找不到 %s" % local_name)
|
||||||
|
return 1
|
||||||
|
upload(str(local), "%s/%s" % (RELAY_DIR, remote_name), mode=0o755)
|
||||||
|
|
||||||
|
print()
|
||||||
|
print(" ── 2. 语法检查 ──")
|
||||||
|
out = run("cd %s && for f in monitor.sh rescue.sh; do "
|
||||||
|
"bash -n $f || exit 1; done && echo SH_OK" % RELAY_DIR,
|
||||||
|
timeout=120, quiet=True)
|
||||||
|
print(" " + out[1].strip())
|
||||||
|
|
||||||
|
print()
|
||||||
|
print(" ── 3. 重置状态 + 跑一次监测 ──")
|
||||||
|
out = run("printf 'fails=0\\nlast_rescue=0\\n' > "
|
||||||
|
"%s/monitor_state.txt; "
|
||||||
|
"bash %s/monitor.sh; echo ' --- 状态 ---'; "
|
||||||
|
"cat %s/monitor_state.txt | sed 's/^/ /'; "
|
||||||
|
"echo ' --- 日志尾 ---'; "
|
||||||
|
"tail -3 %s/monitor.log | sed 's/^/ /'"
|
||||||
|
% (RELAY_DIR, RELAY_DIR, RELAY_DIR, RELAY_DIR),
|
||||||
|
timeout=180, quiet=True)
|
||||||
|
for ln in out[1].strip().splitlines():
|
||||||
|
print(ln)
|
||||||
|
|
||||||
|
print()
|
||||||
|
print(" ── 4. 探测各层现在什么状态 ──")
|
||||||
|
probe = (
|
||||||
|
'printf " %-36s %s\\n" "PRIMARY https://域名" '
|
||||||
|
'"$(curl -sL -o /dev/null -w \'%{http_code}\' --max-time 15 '
|
||||||
|
'https://yk.xxmxxs.top/v1/health)"; '
|
||||||
|
'printf " %-36s %s\\n" "FALLBACK 127.0.0.1:13245" '
|
||||||
|
'"$(curl -s -o /dev/null -w \'%{http_code}\' --max-time 8 '
|
||||||
|
'http://127.0.0.1:13245/v1/health)"; '
|
||||||
|
'printf " %-36s %s\\n" "nginx :80(301 到 https)" '
|
||||||
|
'"$(curl -s -o /dev/null -w \'%{http_code}\' --max-time 8 '
|
||||||
|
'http://127.0.0.1:80/v1/health)"; '
|
||||||
|
'printf " %-36s %s\\n" "nginx :443(本机带 SNI)" '
|
||||||
|
'"$(curl -sk -o /dev/null -w \'%{http_code}\' --max-time 8 '
|
||||||
|
'--resolve yk.xxmxxs.top:443:127.0.0.1 '
|
||||||
|
'https://yk.xxmxxs.top/v1/health)"; '
|
||||||
|
'printf " %-36s %s\\n" "公网 443(Lightsail 防火墙)" '
|
||||||
|
'"$(curl -sk -o /dev/null -w \'%{http_code}\' --max-time 8 '
|
||||||
|
'https://13.54.30.196/v1/health 2>/dev/null || echo 不通)"'
|
||||||
|
)
|
||||||
|
out = run(probe, timeout=180, quiet=True)
|
||||||
|
for ln in out[1].strip().splitlines():
|
||||||
|
print(ln)
|
||||||
|
|
||||||
|
print()
|
||||||
|
print(" 说明:PRIMARY 现在是 522 —— 因为公网 443 还没开。")
|
||||||
|
print(" 这不影响 FALLBACK 正常,所以监测**不会**误报成")
|
||||||
|
print(" '中转挂了';它会把层次标成 edge(问题在边缘层),")
|
||||||
|
print(" 等 443 开了 PRIMARY 自然就 200 了。")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
r'''暂停监测单元,等 443 端口开好再启用。
|
||||||
|
|
||||||
|
== 为什么要暂停 ==
|
||||||
|
|
||||||
|
新架构:用户 -> Cloudflare -> nginx(443) -> 127.0.0.1:13245
|
||||||
|
|
||||||
|
证书签好了、nginx 也配了 443,但 **Lightsail 防火墙还没放行 443**,
|
||||||
|
所以外部访问 https://yk.xxmxxs.top 是 522。
|
||||||
|
|
||||||
|
监测查的是"用户实际走的路",于是一直判失败、fails 一直涨。
|
||||||
|
涨到 3 就启动紧急预案 —— 而预案能做的都解决不了"防火墙没开",
|
||||||
|
结果只会白折腾服务 + 每 10 分钟塞一份没用的报告进后台,
|
||||||
|
把以后真正出事的现场淹掉。
|
||||||
|
|
||||||
|
所以:**系统还没部署完,先别开监测**。443 通了再启用。
|
||||||
|
|
||||||
|
== 暂停不是关掉 ==
|
||||||
|
|
||||||
|
只 stop,配置和脚本都留着,一条命令就能恢复:
|
||||||
|
|
||||||
|
sudo systemctl start ykxmz-monitor.timer
|
||||||
|
|
||||||
|
== 写这个文件的原因 ==
|
||||||
|
|
||||||
|
之前用 `python -c @"..."@` 写过一次,里面 `\$(...)` 转义错了,
|
||||||
|
bash 报 "syntax error near unexpected token `('"。
|
||||||
|
教训一致:**远端命令超过一行就写文件**,别在 -c 里跟转义缠斗。
|
||||||
|
'''
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
sys.path.insert(0, str(ROOT / "_build"))
|
||||||
|
from lssh import run # noqa: E402
|
||||||
|
|
||||||
|
RELAY_DIR = "/opt/ykxmz-relay"
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
print("=" * 62)
|
||||||
|
print(" 暂停监测单元(等 443 开好再启用)")
|
||||||
|
print("=" * 62)
|
||||||
|
|
||||||
|
print()
|
||||||
|
print(" ── 1. 停掉监测定时器 ──")
|
||||||
|
# 分几条简单命令跑,不做复杂嵌套 —— 免得又踩转义
|
||||||
|
run("sudo systemctl stop ykxmz-monitor.timer", timeout=120, quiet=True)
|
||||||
|
out = run("systemctl is-active ykxmz-monitor.timer", timeout=60,
|
||||||
|
quiet=True)
|
||||||
|
print(" monitor.timer = " + out[1].strip())
|
||||||
|
|
||||||
|
print()
|
||||||
|
print(" ── 2. 失败计数清零 ──")
|
||||||
|
run("printf 'fails=0\\nlast_rescue=0\\nlast_code=000\\n"
|
||||||
|
"local_code=200\\nlayer=edge\\n' > %s/monitor_state.txt"
|
||||||
|
% RELAY_DIR, timeout=60, quiet=True)
|
||||||
|
out = run("cat %s/monitor_state.txt" % RELAY_DIR, timeout=60, quiet=True)
|
||||||
|
for ln in out[1].strip().splitlines():
|
||||||
|
print(" " + ln)
|
||||||
|
|
||||||
|
print()
|
||||||
|
print(" ── 3. 清掉演练产生的报告 ──")
|
||||||
|
before = run("ls -1 %s/rescue_reports 2>/dev/null | wc -l" % RELAY_DIR,
|
||||||
|
timeout=60, quiet=True)
|
||||||
|
print(" 清之前: %s 份" % before[1].strip())
|
||||||
|
run("rm -f %s/rescue_reports/*.txt" % RELAY_DIR, timeout=60, quiet=True)
|
||||||
|
after = run("ls -1 %s/rescue_reports 2>/dev/null | wc -l" % RELAY_DIR,
|
||||||
|
timeout=60, quiet=True)
|
||||||
|
print(" 清之后: %s 份" % after[1].strip())
|
||||||
|
|
||||||
|
print()
|
||||||
|
print(" ── 4. 当前状态 ──")
|
||||||
|
st = run("systemctl is-active ykxmz-relay nginx ykxmz-monitor.timer "
|
||||||
|
"certbot.timer", timeout=60, quiet=True)
|
||||||
|
names = ["ykxmz-relay", "nginx", "ykxmz-monitor.timer", "certbot.timer"]
|
||||||
|
for i, ln in enumerate(st[1].strip().splitlines()):
|
||||||
|
if i < len(names):
|
||||||
|
print(" %-24s %s" % (names[i], ln))
|
||||||
|
|
||||||
|
print()
|
||||||
|
code1 = run("curl -s -o /dev/null -w '%{http_code}' "
|
||||||
|
"http://127.0.0.1:13245/v1/health", timeout=60, quiet=True)
|
||||||
|
print(" %-24s %s" % ("中转本机", code1[1].strip()))
|
||||||
|
code2 = run("curl -sk -o /dev/null -w '%{http_code}' --resolve "
|
||||||
|
"yk.xxmxxs.top:443:127.0.0.1 "
|
||||||
|
"https://yk.xxmxxs.top/v1/health", timeout=60, quiet=True)
|
||||||
|
print(" %-24s %s" % ("nginx 443 本机", code2[1].strip()))
|
||||||
|
|
||||||
|
print()
|
||||||
|
print(" 443 开好之后恢复监测:")
|
||||||
|
print(" sudo systemctl start ykxmz-monitor.timer")
|
||||||
|
print(" (或者告诉我,我来开 + 全链路验证)")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,153 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# ============================================================================
|
||||||
|
# 网站监测单元(Lightsail 版)
|
||||||
|
# ============================================================================
|
||||||
|
#
|
||||||
|
# 每 60 秒查一次网站,连续 3 次不通就叫紧急预案。
|
||||||
|
#
|
||||||
|
# ── 为什么查两层 ──────────────────────────────────────────────────────
|
||||||
|
#
|
||||||
|
# PRIMARY https://yk.xxmxxs.top/v1/health
|
||||||
|
# 用户实际走的路:Cloudflare -> nginx -> 中转,全程 https
|
||||||
|
#
|
||||||
|
# FALLBACK http://127.0.0.1:13245/v1/health
|
||||||
|
# 中转本身,绕开 Cloudflare 和 nginx
|
||||||
|
#
|
||||||
|
# 两个都查才能**分清是哪一层坏了**,从而选对抢救动作:
|
||||||
|
#
|
||||||
|
# PRIMARY 挂 + FALLBACK 好 -> 问题在 CF / DNS / nginx 层(edge)
|
||||||
|
# 两个都挂 -> 中转服务本身有问题(core)
|
||||||
|
#
|
||||||
|
# 只查一个的话就没法区分 —— 不管是哪层坏了都只能"重启中转",
|
||||||
|
# 而 edge 层的问题重启中转根本没用。
|
||||||
|
#
|
||||||
|
# ── 为什么用 timer 而不是常驻进程 ──────────────────────────────────────
|
||||||
|
#
|
||||||
|
# 崩了下一分钟自己就回来,不会有"跑久了状态变脏"的问题。
|
||||||
|
#
|
||||||
|
# ── 踩过的两个坑(都写在这里免得再犯)────────────────────────────────
|
||||||
|
#
|
||||||
|
# 1. 改监听地址之后忘了改探测地址
|
||||||
|
# 把中转改成只绑 127.0.0.1 之后,探测还在查公网 :13245,
|
||||||
|
# 于是连续报"网站挂了" -> 启动预案 -> 重启中转 -> 再查还是失败,
|
||||||
|
# 差一点就是无限重启循环。
|
||||||
|
#
|
||||||
|
# 2. 忘了 curl 默认不跟跳转
|
||||||
|
# certbot 加了 http -> https 的 301 之后,探测拿到 301 而不是 200,
|
||||||
|
# 又被判成"挂了"。所以下面用 -L。
|
||||||
|
#
|
||||||
|
# 共同教训:**改了入口的行为,必须回头看所有探测入口的地方。**
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
set -u
|
||||||
|
|
||||||
|
RELAY_DIR="${RELAY_DIR:-/opt/ykxmz-relay}"
|
||||||
|
|
||||||
|
PRIMARY_URL="${PRIMARY_URL:-https://yk.xxmxxs.top}"
|
||||||
|
FALLBACK_URL="${FALLBACK_URL:-http://127.0.0.1:13245}"
|
||||||
|
|
||||||
|
FAIL_THRESHOLD="${FAIL_THRESHOLD:-3}"
|
||||||
|
COOLDOWN="${COOLDOWN:-600}"
|
||||||
|
CHECK_TIMEOUT="${CHECK_TIMEOUT:-10}"
|
||||||
|
|
||||||
|
STATE="$RELAY_DIR/monitor_state.txt"
|
||||||
|
LOG="$RELAY_DIR/monitor.log"
|
||||||
|
LOCKFILE="/tmp/ykxmz-monitor.lock"
|
||||||
|
|
||||||
|
if ! mkdir "$LOCKFILE" 2>/dev/null; then
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
trap 'rmdir "$LOCKFILE" 2>/dev/null' EXIT
|
||||||
|
|
||||||
|
log() { echo "$(date '+%F %T') $*" >> "$LOG"; }
|
||||||
|
|
||||||
|
# ---- 读上次状态 ----
|
||||||
|
FAILS=0
|
||||||
|
LAST_RESCUE=0
|
||||||
|
if [ -f "$STATE" ]; then
|
||||||
|
FAILS=$(grep -m1 '^fails=' "$STATE" 2>/dev/null | cut -d= -f2)
|
||||||
|
LAST_RESCUE=$(grep -m1 '^last_rescue=' "$STATE" 2>/dev/null | cut -d= -f2)
|
||||||
|
FAILS=${FAILS:-0}
|
||||||
|
LAST_RESCUE=${LAST_RESCUE:-0}
|
||||||
|
fi
|
||||||
|
NOW=$(date +%s)
|
||||||
|
|
||||||
|
# ---- 探两层 ----
|
||||||
|
# -L 跟跳转:nginx 那边 http 会 301 到 https。
|
||||||
|
# 不跟跳转的话拿到 301 会被误判成故障(这个坑踩过一次)。
|
||||||
|
PUB_CODE=$(curl -sL -o /dev/null -w '%{http_code}' \
|
||||||
|
--max-time "$CHECK_TIMEOUT" "$PRIMARY_URL/v1/health" 2>/dev/null)
|
||||||
|
PUB_DUR=$(curl -sL -o /dev/null -w '%{time_total}' \
|
||||||
|
--max-time "$CHECK_TIMEOUT" "$PRIMARY_URL/v1/health" 2>/dev/null)
|
||||||
|
LOC_CODE=$(curl -s -o /dev/null -w '%{http_code}' \
|
||||||
|
--max-time "$CHECK_TIMEOUT" "$FALLBACK_URL/v1/health" 2>/dev/null)
|
||||||
|
PUB_CODE=${PUB_CODE:-000}
|
||||||
|
LOC_CODE=${LOC_CODE:-000}
|
||||||
|
PUB_DUR=${PUB_DUR:-0}
|
||||||
|
|
||||||
|
# 健康 = 用户那层通。这层的通才算"网站是好的"。
|
||||||
|
CODE="$PUB_CODE"
|
||||||
|
|
||||||
|
# 哪一层坏了?给预案用的提示
|
||||||
|
LAYER=""
|
||||||
|
if [ "$PUB_CODE" != "200" ] && [ "$LOC_CODE" = "200" ]; then
|
||||||
|
LAYER="edge"
|
||||||
|
elif [ "$PUB_CODE" != "200" ] && [ "$LOC_CODE" != "200" ]; then
|
||||||
|
LAYER="core"
|
||||||
|
fi
|
||||||
|
|
||||||
|
write_state() {
|
||||||
|
{
|
||||||
|
echo "fails=$1"
|
||||||
|
echo "last_rescue=$2"
|
||||||
|
echo "last_code=$PUB_CODE"
|
||||||
|
echo "local_code=$LOC_CODE"
|
||||||
|
echo "layer=$LAYER"
|
||||||
|
echo "last_check=$NOW"
|
||||||
|
echo "last_dur=$PUB_DUR"
|
||||||
|
} > "$STATE"
|
||||||
|
}
|
||||||
|
|
||||||
|
if [ "$CODE" = "200" ]; then
|
||||||
|
PREV=$(grep -m1 '^last_code=' "$STATE" 2>/dev/null | cut -d= -f2)
|
||||||
|
if [ "$PREV" != "200" ] && [ -n "$PREV" ]; then
|
||||||
|
log "网站恢复(HTTP 200,${PUB_DUR}s)—— 之前连续失败 $FAILS 次"
|
||||||
|
fi
|
||||||
|
write_state 0 "$LAST_RESCUE"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
FAILS=$((FAILS + 1))
|
||||||
|
log "网站异常 第 $FAILS 次(用户层 HTTP $PUB_CODE,中转层 HTTP $LOC_CODE${LAYER:+,层次=$LAYER},${PUB_DUR}s)"
|
||||||
|
write_state "$FAILS" "$LAST_RESCUE"
|
||||||
|
|
||||||
|
if [ "$FAILS" -lt "$FAIL_THRESHOLD" ]; then
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
SINCE=$((NOW - LAST_RESCUE))
|
||||||
|
if [ "$LAST_RESCUE" != "0" ] && [ "$SINCE" -lt "$COOLDOWN" ]; then
|
||||||
|
log " 已连续失败 $FAILS 次,但距上次抢救只过了 ${SINCE}s(冷却 ${COOLDOWN}s)"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 知识库提示:历史上哪一级最常奏效
|
||||||
|
HINT=""
|
||||||
|
if [ -f "$RELAY_DIR/rescue_knowledge.md" ]; then
|
||||||
|
HINT=$(grep -oE '第 [0-9] 级奏效' "$RELAY_DIR/rescue_knowledge.md" 2>/dev/null \
|
||||||
|
| sort | uniq -c | sort -rn | head -1 | grep -oE '[0-9]')
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "连续失败 $FAILS 次,启动紧急预案(层次:${LAYER:-未知})${HINT:+,知识库提示第 $HINT 级常奏效}"
|
||||||
|
write_state "$FAILS" "$NOW"
|
||||||
|
|
||||||
|
timeout 300 bash "$RELAY_DIR/rescue.sh" \
|
||||||
|
"监测单元:网站连续 $FAILS 次不通(层次 ${LAYER:-未知})" >> "$LOG" 2>&1
|
||||||
|
RC=$?
|
||||||
|
|
||||||
|
if [ "$RC" = "0" ]; then
|
||||||
|
log " 预案完成:已恢复"
|
||||||
|
else
|
||||||
|
log " 预案完成:仍未恢复(退出码 $RC),报告在 rescue_reports/"
|
||||||
|
fi
|
||||||
|
exit 0
|
||||||
+199
-105
@@ -3,40 +3,54 @@
|
|||||||
# 紧急预案(Lightsail 版)—— 网站打不开时按阶梯自动抢救
|
# 紧急预案(Lightsail 版)—— 网站打不开时按阶梯自动抢救
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
#
|
#
|
||||||
# 和随身 WiFi 那版的区别:
|
# ── 和旧那版的区别 ────────────────────────────────────────────────────
|
||||||
#
|
#
|
||||||
# 那版是**围着 Cloudflare 隧道**转的(第 1 级就是重启隧道、
|
# 旧机器是围着 Cloudflare 隧道转的(第 1 级重启隧道、第 3 级重施
|
||||||
# 第 3 级重施 MSS/MTU 调优)。现在这台新机器**有公网 IP**,
|
# MSS/MTU 调优)。新机器**有公网 IP、没有隧道**,中转跑在 nginx 后面,
|
||||||
# 中转直接对外监听 13245,没有任何隧道 —— 所以:
|
# 所以阶梯完全重写。
|
||||||
#
|
#
|
||||||
# * 第 1 级从"重启隧道"改成"重启中转服务"
|
# ── 核心改进:先看是哪一层坏的 ────────────────────────────────────────
|
||||||
# * 去掉 MSS/MTU 那两级(那是 4G 线路的问题,这里没有)
|
|
||||||
# * 加一级"看端口有没有被占/进程死没死"
|
|
||||||
#
|
#
|
||||||
# 保留下来的核心设计没变:
|
# 监测单元会传进来层次:
|
||||||
|
#
|
||||||
|
# edge = 中转本身是好的(127.0.0.1:13245 返回 200),
|
||||||
|
# 但用户走的路(Cloudflare -> nginx)不通
|
||||||
|
# -> 该动 nginx / 查 DNS / 查 Cloudflare
|
||||||
|
# -> **重启中转服务毫无用处**
|
||||||
|
#
|
||||||
|
# core = 中转本身也不通
|
||||||
|
# -> 该重启中转
|
||||||
|
#
|
||||||
|
# 不区分层次的话就只能瞎重启 —— 而 edge 层的问题重启中转一万次也没用,
|
||||||
|
# 只会把报告淹掉。
|
||||||
|
#
|
||||||
|
# ── 保留的核心设计 ────────────────────────────────────────────────────
|
||||||
#
|
#
|
||||||
# 1. **先记录现场,再动手**(重启会把证据冲掉)
|
# 1. **先记录现场,再动手**(重启会把证据冲掉)
|
||||||
# 2. **阶梯式,每级都验**:从代价最小到最大,好了就停
|
# 2. **阶梯式,每级都验**:代价从最小到最大,好了就停
|
||||||
# 3. **全失败要留证据**:生成 txt 报告,后台可下载
|
# 3. **全失败要留证据**:生成 txt 报告,可下载
|
||||||
# 4. **成功的招数记进知识库**:下次同类问题优先试
|
# 4. **成功的招数记进知识库**:下次同类问题优先试
|
||||||
#
|
#
|
||||||
# 谁调用:monitor.sh(连续 N 次不通就调这里)
|
# 谁调用:monitor.sh
|
||||||
# 也可以人工跑:bash /opt/ykxmz-relay/rescue.sh
|
# 也可以人工跑:bash /opt/ykxmz-relay/rescue.sh "手工测试"
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
|
|
||||||
set -u
|
set -u
|
||||||
|
|
||||||
RELAY_DIR="${RELAY_DIR:-/opt/ykxmz-relay}"
|
RELAY_DIR="${RELAY_DIR:-/opt/ykxmz-relay}"
|
||||||
|
DOMAIN="${DOMAIN:-yk.xxmxxs.top}"
|
||||||
PORT="${RELAY_PORT:-13245}"
|
PORT="${RELAY_PORT:-13245}"
|
||||||
# 公网健康检查地址。没有域名就填 IP。
|
|
||||||
PUBLIC_URL="${PUBLIC_URL:-http://13.54.30.196:$PORT}"
|
PRIMARY_URL="${PRIMARY_URL:-https://$DOMAIN}"
|
||||||
|
FALLBACK_URL="${FALLBACK_URL:-http://127.0.0.1:$PORT}"
|
||||||
|
|
||||||
REPORTS_DIR="$RELAY_DIR/rescue_reports"
|
REPORTS_DIR="$RELAY_DIR/rescue_reports"
|
||||||
KNOWLEDGE="$RELAY_DIR/rescue_knowledge.md"
|
KNOWLEDGE="$RELAY_DIR/rescue_knowledge.md"
|
||||||
LOG="$RELAY_DIR/rescue.log"
|
LOG="$RELAY_DIR/rescue.log"
|
||||||
LOCKFILE="/tmp/ykxmz-rescue.lock"
|
LOCKFILE="/tmp/ykxmz-rescue.lock"
|
||||||
|
|
||||||
SETTLE_FAST=6
|
SETTLE_FAST=6
|
||||||
SETTLE_SLOW=15
|
SETTLE_SLOW=12
|
||||||
|
|
||||||
# 是否允许"重启整机"这张底牌。默认关。
|
# 是否允许"重启整机"这张底牌。默认关。
|
||||||
RESCUE_REBOOT="${RESCUE_REBOOT:-0}"
|
RESCUE_REBOOT="${RESCUE_REBOOT:-0}"
|
||||||
@@ -57,24 +71,34 @@ say() {
|
|||||||
echo "$(date '+%F %T') $*" >> "$LOG"
|
echo "$(date '+%F %T') $*" >> "$LOG"
|
||||||
}
|
}
|
||||||
|
|
||||||
# 公网健康检查 —— **看公网,不看本机**
|
# 用户那层通了才算网站是好的
|
||||||
is_healthy() {
|
is_healthy() {
|
||||||
local code
|
local code
|
||||||
code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 12 \
|
code=$(curl -sL -o /dev/null -w '%{http_code}' --max-time 12 \
|
||||||
"$PUBLIC_URL/v1/health" 2>/dev/null)
|
"$PRIMARY_URL/v1/health" 2>/dev/null)
|
||||||
[ "$code" = "200" ]
|
[ "$code" = "200" ]
|
||||||
}
|
}
|
||||||
|
|
||||||
local_health() {
|
# 中转本身
|
||||||
|
core_ok() {
|
||||||
local code
|
local code
|
||||||
code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 6 \
|
code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 6 \
|
||||||
"http://127.0.0.1:$PORT/v1/health" 2>/dev/null)
|
"$FALLBACK_URL/v1/health" 2>/dev/null)
|
||||||
[ "$code" = "200" ]
|
[ "$code" = "200" ]
|
||||||
}
|
}
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# nginx 本机(带 SNI 走 443)
|
||||||
|
nginx_ok() {
|
||||||
|
local code
|
||||||
|
code=$(curl -sk -o /dev/null -w '%{http_code}' --max-time 8 \
|
||||||
|
--resolve "$DOMAIN:443:127.0.0.1" \
|
||||||
|
"https://$DOMAIN/v1/health" 2>/dev/null)
|
||||||
|
[ "$code" = "200" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
# ===========================================================================
|
||||||
# 现场记录
|
# 现场记录
|
||||||
# ---------------------------------------------------------------------------
|
# ===========================================================================
|
||||||
collect() {
|
collect() {
|
||||||
{
|
{
|
||||||
echo "================================================================================"
|
echo "================================================================================"
|
||||||
@@ -82,74 +106,95 @@ collect() {
|
|||||||
echo " 时间:$(date '+%F %T %Z')"
|
echo " 时间:$(date '+%F %T %Z')"
|
||||||
echo " 主机:$(hostname) 运行时长:$(uptime -p 2>/dev/null || uptime)"
|
echo " 主机:$(hostname) 运行时长:$(uptime -p 2>/dev/null || uptime)"
|
||||||
echo " 触发原因:$1"
|
echo " 触发原因:$1"
|
||||||
|
echo " 层次判断:${LAYER:-未知}"
|
||||||
echo "================================================================================"
|
echo "================================================================================"
|
||||||
echo
|
echo
|
||||||
echo "---- 一、结论速览 ----"
|
echo "---- 一、结论速览 ----"
|
||||||
echo
|
echo
|
||||||
echo " 公网 $PUBLIC_URL/v1/health -> $(is_healthy && echo 200-正常 || echo 不通)"
|
echo " 用户层 $PRIMARY_URL/v1/health -> $(is_healthy && echo '200 正常' || echo 不通)"
|
||||||
echo " 本机 http://127.0.0.1:$PORT/v1/health -> $(local_health && echo 200-正常 || echo 不通)"
|
echo " 中转层 $FALLBACK_URL/v1/health -> $(core_ok && echo '200 正常' || echo 不通)"
|
||||||
|
echo " nginx https://$DOMAIN (本机带 SNI) -> $(nginx_ok && echo '200 正常' || echo 不通)"
|
||||||
echo
|
echo
|
||||||
if local_health && ! is_healthy; then
|
case "${LAYER:-}" in
|
||||||
echo " 判断:**本机是好的,问题在公网那一侧**(防火墙规则、安全组、上游线路)。"
|
edge)
|
||||||
elif ! local_health; then
|
echo " 判断:**中转是好的,问题在边缘层**(Cloudflare / DNS / nginx)。"
|
||||||
echo " 判断:**本机的中转服务就没起来**,先看第二、三节。"
|
echo " 要动的是 nginx,不是中转。"
|
||||||
else
|
echo " 常见原因:"
|
||||||
echo " 判断:两边都通(可能是刚才已经自愈,或者只是偶发抖动)。"
|
echo " * Lightsail 防火墙没开 443(Cloudflare Full 模式回源要用)"
|
||||||
fi
|
echo " * Cloudflare 的 SSL/TLS 模式和源站对不上"
|
||||||
|
echo " * DNS 记录被改成了灰色云朵,或者指向了别的 IP"
|
||||||
|
;;
|
||||||
|
core)
|
||||||
|
echo " 判断:**中转服务本身不通**,先看第二、三节。"
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo " 判断:两层都通(可能已自愈),或者只是偶发抖动。"
|
||||||
|
;;
|
||||||
|
esac
|
||||||
echo
|
echo
|
||||||
echo "---- 二、服务状态 ----"
|
echo "---- 二、服务状态 ----"
|
||||||
echo
|
echo
|
||||||
printf ' %-24s %s\n' "ykxmz-relay" "$(systemctl is-active ykxmz-relay 2>&1)"
|
for s in ykxmz-relay nginx ykxmz-monitor.timer certbot.timer; do
|
||||||
printf ' %-24s %s\n' "ykxmz-monitor.timer" "$(systemctl is-active ykxmz-monitor.timer 2>&1)"
|
printf ' %-24s %s\n' "$s" "$(systemctl is-active $s 2>&1)"
|
||||||
|
done
|
||||||
echo
|
echo
|
||||||
echo " --- 最近 20 行服务日志 ---"
|
echo " --- 中转最近 15 行日志 ---"
|
||||||
journalctl -u ykxmz-relay --no-pager -n 20 2>/dev/null | sed 's/^/ /'
|
journalctl -u ykxmz-relay --no-pager -n 15 2>/dev/null | sed 's/^/ /'
|
||||||
|
echo
|
||||||
|
echo " --- nginx 最近 15 行错误日志 ---"
|
||||||
|
tail -15 /var/log/nginx/ykxmz-error.log 2>/dev/null | sed 's/^/ /'
|
||||||
echo
|
echo
|
||||||
echo "---- 三、进程与端口 ----"
|
echo "---- 三、进程与端口 ----"
|
||||||
echo
|
echo
|
||||||
echo " 监听 $PORT 的进程:"
|
echo " 监听情况:"
|
||||||
ss -tlnp 2>/dev/null | grep ":$PORT " | sed 's/^/ /' \
|
ss -tlnp 2>/dev/null | grep -E ':(80|443|'"$PORT"') ' | sed 's/^/ /'
|
||||||
|| echo " **没有任何进程在监听 $PORT**"
|
|
||||||
echo
|
echo
|
||||||
echo " 相关进程:"
|
echo " 相关进程:"
|
||||||
ps -eo pid,rss,etime,cmd --sort=-rss 2>/dev/null \
|
ps -eo pid,rss,etime,cmd --sort=-rss 2>/dev/null \
|
||||||
| grep -E 'relay\.py|python3' | grep -v grep | head -6 | sed 's/^/ /'
|
| grep -E 'relay\.py|nginx' | grep -v grep | head -6 | sed 's/^/ /'
|
||||||
echo
|
echo
|
||||||
echo "---- 四、系统资源 ----"
|
echo "---- 四、本机各层自测 ----"
|
||||||
|
echo
|
||||||
|
printf ' %-34s %s\n' "中转 127.0.0.1:$PORT" \
|
||||||
|
"$(curl -s -o /dev/null -w '%{http_code}' --max-time 6 http://127.0.0.1:$PORT/v1/health)"
|
||||||
|
printf ' %-34s %s\n' "nginx :80(应 301)" \
|
||||||
|
"$(curl -s -o /dev/null -w '%{http_code}' --max-time 6 http://127.0.0.1:80/v1/health)"
|
||||||
|
printf ' %-34s %s\n' "nginx :443(带 SNI)" \
|
||||||
|
"$(curl -sk -o /dev/null -w '%{http_code}' --max-time 8 --resolve "$DOMAIN:443:127.0.0.1" https://$DOMAIN/v1/health)"
|
||||||
|
printf ' %-34s %s\n' "用户层 $PRIMARY_URL" \
|
||||||
|
"$(curl -sL -o /dev/null -w '%{http_code}' --max-time 12 $PRIMARY_URL/v1/health)"
|
||||||
|
echo
|
||||||
|
echo "---- 五、证书 ----"
|
||||||
|
echo
|
||||||
|
certbot certificates 2>/dev/null | grep -E 'Certificate Name|Domains|Expiry' \
|
||||||
|
| sed 's/^/ /'
|
||||||
|
echo
|
||||||
|
echo "---- 六、系统资源 ----"
|
||||||
echo
|
echo
|
||||||
free -m 2>/dev/null | sed 's/^/ /'
|
free -m 2>/dev/null | sed 's/^/ /'
|
||||||
echo
|
echo
|
||||||
uptime | sed 's/^/ /'
|
uptime | sed 's/^/ /'
|
||||||
echo
|
echo
|
||||||
df -h / /opt 2>/dev/null | sed 's/^/ /'
|
df -h / 2>/dev/null | sed 's/^/ /'
|
||||||
echo
|
echo
|
||||||
echo "---- 五、网络 ----"
|
echo "---- 七、网络 ----"
|
||||||
echo
|
echo
|
||||||
ip -brief addr show 2>/dev/null | sed 's/^/ /'
|
ip -brief addr show 2>/dev/null | sed 's/^/ /'
|
||||||
echo
|
echo
|
||||||
echo " 默认路由:"
|
echo " 默认路由:"
|
||||||
ip route show default 2>/dev/null | sed 's/^/ /'
|
ip route show default 2>/dev/null | sed 's/^/ /'
|
||||||
echo
|
echo
|
||||||
echo " 本机自测各端口:"
|
echo " 公网 IP:$(curl -s --max-time 8 https://api.ipify.org 2>/dev/null || echo 取不到)"
|
||||||
for p in "$PORT" 80 443 22; do
|
|
||||||
printf ' %-6s ' "$p"
|
|
||||||
timeout 3 bash -c "</dev/tcp/127.0.0.1/$p" 2>/dev/null \
|
|
||||||
&& echo "通" || echo "不通"
|
|
||||||
done
|
|
||||||
echo
|
echo
|
||||||
echo "---- 六、日志尾部 ----"
|
echo " DNS 解析 $DOMAIN:"
|
||||||
|
getent hosts "$DOMAIN" 2>/dev/null | sed 's/^/ /' || echo " 解析失败"
|
||||||
echo
|
echo
|
||||||
echo "### 中转服务 relay.log"
|
echo "---- 八、内核异常(OOM / 硬件)----"
|
||||||
tail -30 "$RELAY_DIR/relay.log" 2>/dev/null | sed 's/^/ /'
|
|
||||||
echo
|
echo
|
||||||
echo "### 监测单元 monitor.log"
|
dmesg -T 2>/dev/null | grep -i -E 'oom|killed process|panic' \
|
||||||
tail -20 "$RELAY_DIR/monitor.log" 2>/dev/null | sed 's/^/ /'
|
| tail -10 | sed 's/^/ /'
|
||||||
echo
|
echo
|
||||||
echo "### 内核异常(OOM / 硬件)"
|
echo "---- 九、最近一次启动 ----"
|
||||||
dmesg -T 2>/dev/null \
|
|
||||||
| grep -i -E 'oom|killed process|error|panic' | tail -15 | sed 's/^/ /'
|
|
||||||
echo
|
|
||||||
echo "---- 七、最近一次启动 ----"
|
|
||||||
echo
|
echo
|
||||||
who -b 2>/dev/null | sed 's/^/ /'
|
who -b 2>/dev/null | sed 's/^/ /'
|
||||||
echo " 运行时长:$(cut -d. -f1 /proc/uptime 2>/dev/null) 秒"
|
echo " 运行时长:$(cut -d. -f1 /proc/uptime 2>/dev/null) 秒"
|
||||||
@@ -161,53 +206,73 @@ collect() {
|
|||||||
say " 现场已记录 -> $REPORT"
|
say " 现场已记录 -> $REPORT"
|
||||||
}
|
}
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ===========================================================================
|
||||||
# 阶梯
|
# 阶梯 —— 按层次走
|
||||||
# ---------------------------------------------------------------------------
|
# ===========================================================================
|
||||||
|
|
||||||
step_1_restart_relay() {
|
step_edge_1_reload_nginx() {
|
||||||
say " [1/4] 重启中转服务(最轻的一招,不动系统其他部分)"
|
say " [edge-1] 重载 nginx 配置(最轻,不断连接)"
|
||||||
systemctl restart ykxmz-relay 2>/dev/null
|
sudo nginx -t >/dev/null 2>&1 && sudo systemctl reload nginx 2>/dev/null
|
||||||
sleep "$SETTLE_FAST"
|
sleep "$SETTLE_FAST"
|
||||||
}
|
}
|
||||||
|
|
||||||
step_2_kill_and_start() {
|
step_edge_2_restart_nginx() {
|
||||||
say " [2/4] 强杀残留进程再拉起(进程卡死/端口被占时用)"
|
say " [edge-2] 重启 nginx(重载不够时用)"
|
||||||
# 有时候 systemd restart 会因为老进程没退干净而失败,
|
sudo systemctl restart nginx 2>/dev/null
|
||||||
# 表现为"服务说 active 但端口没人听"。
|
|
||||||
pkill -f "$RELAY_DIR/relay.py" 2>/dev/null
|
|
||||||
sleep 3
|
|
||||||
systemctl reset-failed ykxmz-relay 2>/dev/null
|
|
||||||
systemctl start ykxmz-relay 2>/dev/null
|
|
||||||
sleep "$SETTLE_FAST"
|
sleep "$SETTLE_FAST"
|
||||||
}
|
}
|
||||||
|
|
||||||
step_3_check_firewall() {
|
step_edge_3_check_443() {
|
||||||
say " [3/4] 检查本机防火墙有没有挡住 $PORT"
|
say " [edge-3] 查公网 443 通不通(Cloudflare Full 模式回源要用)"
|
||||||
# Lightsail 有两层:网页上的防火墙规则(实例级)+ 系统内的 ufw/iptables。
|
if timeout 5 bash -c "</dev/tcp/$(curl -s --max-time 6 https://api.ipify.org 2>/dev/null)/443" 2>/dev/null; then
|
||||||
# 网页那层我们改不了,但系统内这层能看。
|
say " 本机自连 443 通"
|
||||||
if command -v ufw >/dev/null 2>&1; then
|
else
|
||||||
echo " ufw 状态:$(ufw status 2>/dev/null | head -1)" >> "$LOG"
|
say " ⚠ 本机自连 443 **不通** —— nginx 可能没在 443 上"
|
||||||
ufw status 2>/dev/null | grep -q "$PORT" \
|
ss -tln 2>/dev/null | grep ':443 ' | sed 's/^/ /'
|
||||||
|| say " ⚠ ufw 里没有放行 $PORT 的规则"
|
|
||||||
fi
|
fi
|
||||||
local n
|
say " ⚠ 提醒:**Lightsail 网页上的防火墙规则**系统内部改不了。"
|
||||||
n=$(iptables -L INPUT -n 2>/dev/null | grep -c "$PORT")
|
say " 如果这里一切正常但公网还是不通,去网页上确认 443 已放行。"
|
||||||
say " iptables INPUT 里提到 $PORT 的规则:$n 条"
|
# 顺手确认证书还在
|
||||||
# 重新绑一次端口(有时是监听 socket 坏了)
|
if [ -f "/etc/letsencrypt/live/$DOMAIN/fullchain.pem" ]; then
|
||||||
systemctl restart ykxmz-relay 2>/dev/null
|
say " 证书存在,到期日:$(openssl x509 -in "/etc/letsencrypt/live/$DOMAIN/fullchain.pem" -noout -enddate 2>/dev/null | cut -d= -f2)"
|
||||||
|
else
|
||||||
|
say " ⚠ **找不到证书**!/etc/letsencrypt/live/$DOMAIN/fullchain.pem"
|
||||||
|
fi
|
||||||
|
sleep "$SETTLE_SLOW"
|
||||||
|
}
|
||||||
|
|
||||||
|
step_core_1_restart_relay() {
|
||||||
|
say " [core-1] 重启中转服务"
|
||||||
|
sudo systemctl restart ykxmz-relay 2>/dev/null
|
||||||
|
sleep "$SETTLE_FAST"
|
||||||
|
}
|
||||||
|
|
||||||
|
step_core_2_kill_and_start() {
|
||||||
|
say " [core-2] 强杀残留进程再拉起(进程卡死/端口被占时用)"
|
||||||
|
sudo pkill -f "$RELAY_DIR/relay.py" 2>/dev/null
|
||||||
|
sleep 3
|
||||||
|
sudo systemctl reset-failed ykxmz-relay 2>/dev/null
|
||||||
|
sudo systemctl start ykxmz-relay 2>/dev/null
|
||||||
|
sleep "$SETTLE_FAST"
|
||||||
|
}
|
||||||
|
|
||||||
|
step_core_3_restart_all() {
|
||||||
|
say " [core-3] 中转 + nginx 都重启一遍"
|
||||||
|
sudo systemctl restart ykxmz-relay 2>/dev/null
|
||||||
|
sleep 2
|
||||||
|
sudo systemctl restart nginx 2>/dev/null
|
||||||
sleep "$SETTLE_SLOW"
|
sleep "$SETTLE_SLOW"
|
||||||
}
|
}
|
||||||
|
|
||||||
step_4_reboot() {
|
step_4_reboot() {
|
||||||
if [ "$RESCUE_REBOOT" != "1" ]; then
|
if [ "$RESCUE_REBOOT" != "1" ]; then
|
||||||
say " [4/4] 重启整机 —— **已跳过**(默认不开)"
|
say " [4] 重启整机 —— **已跳过**(默认不开)"
|
||||||
say " 要允许自动重启,把 RESCUE_REBOOT=1 写进 ykxmz-monitor.service。"
|
say " 要允许自动重启,把 RESCUE_REBOOT=1 写进 ykxmz-monitor.service。"
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
say " [4/4] 重启整机(最后的底牌)"
|
say " [4] 重启整机(最后的底牌)"
|
||||||
sync
|
sync
|
||||||
systemctl reboot 2>/dev/null
|
sudo systemctl reboot 2>/dev/null
|
||||||
return 0
|
return 0
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -218,8 +283,9 @@ remember() {
|
|||||||
echo "## $(date '+%F %T') 第 $step 级奏效"
|
echo "## $(date '+%F %T') 第 $step 级奏效"
|
||||||
echo
|
echo
|
||||||
echo "- 症状:$reason"
|
echo "- 症状:$reason"
|
||||||
|
echo "- 层次:${LAYER:-未知}"
|
||||||
echo "- 有效步骤:$step"
|
echo "- 有效步骤:$step"
|
||||||
echo "- 当时公网:$(is_healthy && echo 通 || echo 不通);本机:$(local_health && echo 通 || echo 不通)"
|
echo "- 当时用户层:$(is_healthy && echo 通 || echo 不通);中转层:$(core_ok && echo 通 || echo 不通)"
|
||||||
echo "- 详情见 rescue_reports/$(basename "$REPORT")"
|
echo "- 详情见 rescue_reports/$(basename "$REPORT")"
|
||||||
} >> "$KNOWLEDGE"
|
} >> "$KNOWLEDGE"
|
||||||
say " 已把「第 $step 级奏效」记进 $KNOWLEDGE"
|
say " 已把「第 $step 级奏效」记进 $KNOWLEDGE"
|
||||||
@@ -231,49 +297,77 @@ remember() {
|
|||||||
|
|
||||||
REASON="${1:-monitor 触发}"
|
REASON="${1:-monitor 触发}"
|
||||||
|
|
||||||
|
# 从触发原因里认出层次(monitor.sh 会传"层次 edge"这样的字样)
|
||||||
|
LAYER=""
|
||||||
|
case "$REASON" in
|
||||||
|
*"层次 edge"*) LAYER="edge" ;;
|
||||||
|
*"层次 core"*) LAYER="core" ;;
|
||||||
|
esac
|
||||||
|
# 认不出来就自己判
|
||||||
|
if [ -z "$LAYER" ]; then
|
||||||
|
if ! core_ok; then LAYER="core"
|
||||||
|
elif ! is_healthy; then LAYER="edge"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
say ""
|
say ""
|
||||||
say "================================================================================"
|
say "================================================================================"
|
||||||
say " 紧急预案启动:$REASON"
|
say " 紧急预案启动:$REASON"
|
||||||
|
say " 层次判断:${LAYER:-两层都通}"
|
||||||
say "================================================================================"
|
say "================================================================================"
|
||||||
|
|
||||||
# ⚠ 先复查再记录现场。
|
# ⚠ 先复查再记录现场。
|
||||||
# collect() 会写一份报告文件;如果网站其实已经好了,
|
# collect() 会写一份报告文件;如果网站其实已经好了,
|
||||||
# 那一次也会留下报告,把真正出事时的报告淹掉(后台只显示最近 30 份)。
|
# 那一次也会留下报告,把真正出事时的报告淹掉(后台只显示最近 30 份)。
|
||||||
if is_healthy; then
|
if is_healthy; then
|
||||||
say " 复查:公网现在是通的,不需要动手 —— 不采现场、不留报告。"
|
say " 复查:用户层现在是通的,不需要动手 —— 不采现场、不留报告。"
|
||||||
exit 0
|
exit 0
|
||||||
fi
|
fi
|
||||||
|
|
||||||
collect "$REASON"
|
collect "$REASON"
|
||||||
|
|
||||||
STEP_NAMES=("重启中转服务" "强杀残留再拉起" "查防火墙并重绑" "重启整机")
|
if [ "$LAYER" = "edge" ]; then
|
||||||
|
STEPS=("重载 nginx" "重启 nginx" "查 443 与证书")
|
||||||
|
FUNCS=(step_edge_1_reload_nginx step_edge_2_restart_nginx step_edge_3_check_443)
|
||||||
|
else
|
||||||
|
STEPS=("重启中转服务" "强杀残留再拉起" "中转+nginx 都重启")
|
||||||
|
FUNCS=(step_core_1_restart_relay step_core_2_kill_and_start step_core_3_restart_all)
|
||||||
|
fi
|
||||||
|
|
||||||
for i in 1 2 3 4; do
|
for i in 0 1 2; do
|
||||||
case $i in
|
${FUNCS[$i]}
|
||||||
1) step_1_restart_relay ;;
|
|
||||||
2) step_2_kill_and_start ;;
|
|
||||||
3) step_3_check_firewall ;;
|
|
||||||
4) step_4_reboot || continue ;;
|
|
||||||
esac
|
|
||||||
|
|
||||||
if is_healthy; then
|
if is_healthy; then
|
||||||
say ""
|
say ""
|
||||||
say " ✅ 第 $i 级(${STEP_NAMES[$((i-1))]})救回来了,公网已恢复。"
|
say " ✅ 第 $((i+1)) 级(${STEPS[$i]})救回来了,用户层已恢复。"
|
||||||
remember "$i" "$REASON"
|
remember "$((i+1))" "$REASON"
|
||||||
exit 0
|
exit 0
|
||||||
fi
|
fi
|
||||||
say " 第 $i 级没解决,继续下一级"
|
say " 第 $((i+1)) 级没解决,继续下一级"
|
||||||
done
|
done
|
||||||
|
|
||||||
|
# 兜底:把两边都试一遍(有时候层次判错了)
|
||||||
say ""
|
say ""
|
||||||
say " ❌ 四级都没解决。再采一次现场(状态已经变了)"
|
say " 三轮都没解决,兜底:中转和 nginx 各重启一次"
|
||||||
collect "$REASON(四级抢救后仍不通)"
|
sudo systemctl restart ykxmz-relay 2>/dev/null
|
||||||
|
sleep 3
|
||||||
|
sudo systemctl restart nginx 2>/dev/null
|
||||||
|
sleep "$SETTLE_SLOW"
|
||||||
|
if is_healthy; then
|
||||||
|
say " ✅ 兜底重启救回来了"
|
||||||
|
remember "兜底" "$REASON"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
say ""
|
||||||
|
say " ❌ 全部没解决。再采一次现场(状态已经变了)"
|
||||||
|
collect "$REASON(多轮抢救后仍不通)"
|
||||||
|
|
||||||
{
|
{
|
||||||
echo "⚠ 这份报告是「自动抢救全部失败」之后生成的。"
|
echo "⚠ 这份报告是「自动抢救全部失败」之后生成的。"
|
||||||
echo " 机器自己试过了:重启中转 / 强杀重拉 / 查防火墙。"
|
echo " 机器自己试过了:重载/重启 nginx、重启中转、强杀重拉。"
|
||||||
echo " (重启整机默认没做。)"
|
echo " (重启整机默认没做。)"
|
||||||
echo " 需要人工看一眼 —— 特别是 Lightsail **网页上的防火墙规则**,"
|
echo " 需要人工看一眼 —— 特别是 **Lightsail 网页上的防火墙规则**,"
|
||||||
echo " 那一层系统内部改不了。"
|
echo " 那一层系统内部改不了。"
|
||||||
echo
|
echo
|
||||||
cat "$REPORT"
|
cat "$REPORT"
|
||||||
|
|||||||
Reference in New Issue
Block a user