错误 #2822
16P_Pre1T2压力测试中，核心网从82切换到80
0%
描述
系统压力测试过程中发生核心网从82切换到80，测试时长大概2天，AMF模块挂死。
核心网版本3.2.3pre7。在这个版本上偶现1次，后面继续观察。
文件
历史记录
由 徐 红超 更新于 3 个月 之前
fatal error: concurrent map read and map write
goroutine 1758525 [running]:
runtime.throw(0x1138901, 0x21)
#011/usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc0005e70f8 sp=0xc0005e70c8 pc=0x433372
runtime.mapaccess1_faststr(0xf9aae0, 0xc000434510, 0x1120eb9, 0xb, 0x1)
#011/usr/local/go/src/runtime/map_faststr.go:21 +0x43c fp=0xc0005e7168 sp=0xc0005e70f8 pc=0x4129fc
github.com/free5gc/amf/internal/sbi/producer.N1N2MessageTransferProcedure(0xc0006e40da, 0x14, 0xc0006e40c0, 0x3d, 0xc0000bc5b0, 0xc00021cc00, 0x45, 0x3e8, 0xc00021d000, 0x2f, ...)
#011/home/ngc/go/src/free5gc/NFs/amf/internal/sbi/producer/n1n2message.go:229 +0xf7a fp=0xc0005e76d8 sp=0xc0005e7168 pc=0xe340fa
github.com/free5gc/amf/internal/sbi/producer.HandleN1N2MessageTransferRequest(0xc000434c30, 0xc0003a8500, 0x111c83a, 0x6)
#011/home/ngc/go/src/free5gc/NFs/amf/internal/sbi/producer/n1n2message.go:30 +0x219 fp=0xc0005e7840 sp=0xc0005e76d8 pc=0xe32c39
github.com/free5gc/amf/internal/sbi/communication.HTTPN1N2MessageTransfer(0xc0003a8500)
由 徐 红超 更新于 3 个月 之前
amflog:
time="2025-02-17T03:50:03+08:00" level=info msg="Select SMF [snssai: {Sst:1 Sd:010203}, dnn: internet]" amf_ue_ngap_id="AMF_UE_NGAP_ID:42319" category=GMM component=AMF supi="SUPI:imsi-001010000000005"
time="2025-02-17T03:50:03+08:00" level=info msg="create smContext[pduSessionID: 6] Success" amf_ue_ngap_id="AMF_UE_NGAP_ID:42319" category=GMM component=AMF supi="SUPI:imsi-001010000000005"
time="2025-02-17T03:50:03+08:00" level=info msg="Handle UE Context Release Request" amf_ue_ngap_id="AMF_UE_NGAP_ID:42319" category=NGAP component=AMF ran_addr="192.168.2.176:38412"
time="2025-02-17T03:50:03+08:00" level=warning msg="Cause RadioNetwork20" category=NGAP component=AMF ran_addr="192.168.2.176:38412"
time="2025-02-17T03:50:03+08:00" level=info msg="Ue Context in GMM-Registered" amf_ue_ngap_id="AMF_UE_NGAP_ID:42319" category=NGAP component=AMF ran_addr="192.168.2.176:38412"
time="2025-02-17T03:50:03+08:00" level=info msg="Ue Context in GMM-Registered,pDUSessionResourceList is nil" amf_ue_ngap_id="AMF_UE_NGAP_ID:42319" category=NGAP component=AMF ran_addr="192.168.2.176:38412"
time="2025-02-17T03:50:03+08:00" level=info msg="**Handle N1N2 Message Transfer Request" category=Producer component=AMF
由 徐 红超 更新于 3 个月 之前
修复该问题后：出现其它挂死地方：
[Switching to LWP 7348]
0x0000000000e28c8d in github.com/free5gc/amf/internal/gmm.HandleServiceRequest.func1 (key=..., value=..., ~r2=<optimized out>)
at /home/ngc/go/src/free5gc/NFs/amf/internal/gmm/handler.go:1739
1739 /home/ngc/go/src/free5gc/NFs/amf/internal/gmm/handler.go: No such file or directory.
(gdb) bt
#0 0x0000000000e28c8d in github.com/free5gc/amf/internal/gmm.HandleServiceRequest.func1 (key=..., value=..., ~r2=<optimized out>)
at /home/ngc/go/src/free5gc/NFs/amf/internal/gmm/handler.go:1739
#1 0x000000000047bc45 in sync.(Map).Range (m=0xc000018960, f={void (interface {}, interface {}, bool *)} 0xc0000a5360) at /usr/local/go/src/sync/map.go:333
#2 0x0000000000e1c16a in github.com/free5gc/amf/internal/gmm.HandleServiceRequest (ue=0xc000018600, anType=..., serviceRequest=0xc000719b00, ~r3=...)
at /home/ngc/go/src/free5gc/NFs/amf/internal/gmm/handler.go:1704
#3 0x0000000000e235e8 in github.com/free5gc/amf/internal/gmm.Registered (state=0xc0003634a0, event=..., args=0xc000719b60)
at /home/ngc/go/src/free5gc/NFs/amf/internal/gmm/sm.go:96
#4 0x0000000000a67d36 in github.com/free5gc/util/fsm.(*FSM).SendEvent (fsm=0xc000512060, state=0xc0003634a0, event=..., args=0xc000719b60, ~r3=...)
at /home/ngc/go/pkg/mod/github.com/free5gc/util@v1.0.3/fsm/fsm.go:98
#5 0x0000000000e29f9e in github.com/free5gc/amf/internal/nas.Dispatch (ue=0xc000018600, accessType=..., procedureCode=46, msg=0xc000386240, ~r4=...)
at /home/ngc/go/src/free5gc/NFs/amf/internal/nas/dispatch.go:27
#6 0x0000000000e2a2a9 in github.com/free5gc/amf/internal/nas.HandleNAS (ue=0xc00018c480, procedureCode=46, nasPdu=...)
at /home/ngc/go/src/free5gc/NFs/amf/internal/nas/handler.go:33
#7 0x0000000000e45ae4 in github.com/free5gc/amf/internal/ngap.HandleUplinkNasTransport (ran=0xc0005ee000, message=0xc000520f40)
at /home/ngc/go/src/free5gc/NFs/amf/internal/ngap/handler.go:241
#8 0x0000000000e43260 in github.com/free5gc/amf/internal/ngap.Dispatch (conn=..., msg=...) at /home/ngc/go/src/free5gc/NFs/amf/internal/ngap/dispatcher.go:61
#9 0x0000000000e6edb0 in github.com/free5gc/amf/internal/ngap/service.handleConnection (conn=0xc0005aa040, bufsize=8192, handler=...)
at /home/ngc/go/src/free5gc/NFs/amf/internal/ngap/service/service.go:206
#10 0x0000000000463421 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1373
#11 0x000000c0005aa040 in ?? ()
#12 0x0000000000002000 in ?? ()
#13 0x00000000011614e0 in func. ()
#14 0x00000000011614f8 in func.* ()
#15 0x0000000000000000 in ?? ()
由 徐 红超 更新于 3 个月 之前
amflog:
time="2025-02-20T14:41:29+08:00" level=warning msg="Parse AMBR failed strconv.ParseFloat: parsing \"2Gbps\": invalid syntax" category=NGAP component=LIB
time="2025-02-20T14:41:29+08:00" level=warning msg="Parse AMBR failed strconv.ParseFloat: parsing \"2Gbps\": invalid syntax" category=NGAP component=LIB
time="2025-02-20T14:41:29+08:00" level=info msg="Handle Initial Context Setup Response" amf_ue_ngap_id="AMF_UE_NGAP_ID:2120" category=NGAP component=AMF ran_addr="192.168.2.176:38412"
time="2025-02-20T14:41:29+08:00" level=info msg="[OAM] Handle Registered gnb Context" category=Producer component=AMF
time="2025-02-20T14:41:29+08:00" level=info msg="| 200 | 192.168.2.139 | GET | /namf-oam/v1/registered-gnb-context | " category=GIN component=AMF
time="2025-02-20T14:41:29+08:00" level=info msg="[OAM] Handle Registered UE Context" category=Producer component=AMF
time="2025-02-20T14:41:29+08:00" level=info msg="| 200 | 192.168.2.139 | GET | /namf-oam/v1/registered-ue-context | " category=GIN component=AMF
time="2025-02-20T14:41:29+08:00" level=info msg="Handle UE Context Release Complete" amf_ue_ngap_id="AMF_UE_NGAP_ID:2119" category=NGAP component=AMF ran_addr="192.168.2.176:38412"
time="2025-02-20T14:41:29+08:00" level=info msg="Rel Ue Context in GMM-Registered" amf_ue_ngap_id="AMF_UE_NGAP_ID:2119" category=NGAP component=AMF ran_addr="192.168.2.176:38412"
time="2025-02-20T14:41:29+08:00" level=info msg="Release UE[imsi-001010000000005] Context : N2 Connection Release" amf_ue_ngap_id="AMF_UE_NGAP_ID:2119" category=NGAP component=AMF ran_addr="192.168.2.176:38412"
time="2025-02-20T14:41:30+08:00" level=info msg="Uplink NAS Transport (RAN UE NGAP ID: 264)" amf_ue_ngap_id="AMF_UE_NGAP_ID:2120" category=NGAP component=AMF ran_addr="192.168.2.176:38412"
time="2025-02-20T14:41:30+08:00" level=info msg="Handle event[Gmm Message], transition from [Registered] to [Registered]" category=FSM component=LIB
time="2025-02-20T14:41:30+08:00" level=info msg="Handle Service Request" category=GMM component=AMF supi="SUPI:imsi-001010000000005"
