Hi,
in our solution we use RabbitMQ server. Frames are transported with use MQTT and AMQP depends on device. In last 2 months 2 times we noticed errors with AMQP.
To fix this we had to restart chirpstack-application-server. But in production it is not possible to always fix it in these way. Could you help me?
I saw that in v3.8.1 * Fix AMQP re-connect issue.
… but I am not sure that this bug was about same
network-server: 3.8.0
application-server: 3.9.0
RabbitMQ: 3.8.2 + Erlang 22.0.7
Output from journalctl:
May 17 11:24:19 loranet-srv-01.emitel.local chirpstack-application-server[32693]: time=“2021-05-17T11:24:19+02:00” level=error msg=“integration/multi: integration error” ctx_id=3c2a7651-03f0-4f34-9ac5-bbeaefce92f0 error=“publish event error: Exception (504) Reason: "channel/connection is not open"” integration=“*amqp.Integration”
logs from RabbitMQ crash log:
2021-06-03 03:59:06.403 [info] <0.25055.366> connection <0.25055.366> (127.0.0.1:59682 → 127.0.0.1:5672): user ‘logstash’ authenticated and granted access to vhost ‘/’
2021-06-03 03:59:06.409 [info] <0.24970.366> connection <0.24970.366> (127.0.0.1:59684 → 127.0.0.1:5672): user ‘logstash’ authenticated and granted access to vhost ‘/’
2021-06-03 04:01:17.436 [warning] <0.32299.304> closing AMQP connection <0.32299.304> (10.93.0.1:60538 → 10.93.0.2:5672, vhost: ‘/’, user: ‘chirpstack_as’):
client unexpectedly closed TCP connection
2021-06-03 04:01:18.517 [info] <0.25425.366> MQTT vhost picked using plugin configuration or default
2021-06-03 04:01:18.711 [warning] <0.24203.212> MQTT disconnecting client “172.20.102.1:60372 → 10.93.0.2:1883” with duplicate id ‘logger-1334’
2021-06-03 04:01:18.712 [info] <0.25425.366> accepting MQTT connection <0.25425.366> (172.20.102.1:51255 → 10.93.0.2:1883, client id: logger-1334)
2021-06-03 04:01:24.927 [info] <0.25471.366> MQTT vhost picked using plugin configuration or default
2021-06-03 04:01:26.472 [warning] <0.24249.212> MQTT disconnecting client “172.20.102.1:60371 → 10.93.0.2:1883” with duplicate id ‘logger-2755’
2021-06-03 04:01:26.474 [info] <0.25471.366> accepting MQTT connection <0.25471.366> (172.20.102.1:51297 → 10.93.0.2:1883, client id: logger-2755)
2021-06-03 04:01:30.096 [info] <0.24814.366> MQTT vhost picked using plugin configuration or default
2021-06-03 04:01:31.290 [error] <0.32645.304> MQTT: a socket write failed, the socket might already be closed
2021-06-03 04:01:31.290 [info] <0.32645.304> MQTT connection “10.93.0.1:44382 → 10.93.0.2:1883” will terminate because peer closed TCP connection
2021-06-03 04:01:35.104 [error] <0.24814.366> MQTT cannot accept a connection: client ID registration timed out
2021-06-03 04:01:35.104 [error] <0.24814.366> MQTT cannot accept connection 10.93.0.1:35330 → 10.93.0.2:1883 due to an internal error or unavailable component
2021-06-03 04:01:36.291 [error] <0.32645.304> ** Generic server <0.32645.304> terminating
** Last message in was {tcp_closed,#Port<0.1814243>}
** When Server state == {state,#Port<0.1814243>,“10.93.0.1:44382 → 10.93.0.2:1883”,true,undefined,true,running,{none,<0.32296.304>},<0.32638.304>,false,none,{proc_state,#Port<0.1814243>,#{“application/+/device/+/tx” => [0]},{<<“amq.ctag
-YOowwvGDd4Gio7wHkLEpDQ”>>,undefined},{0,nil},{0,nil},undefined,1,“AckMrCeAAx2bHcmZcnfOzg”,true,undefined,{<0.32618.304>,undefined},<0.32622.304>,<<“amq.topic”>>,{amqp_adapter_info,{0,0,0,0,0,65535,2653,2},1883,{0,0,0,0,0,65535,2653,1},4
4382,<<“10.93.0.1:44382 → 10.93.0.2:1883”>>,{‘MQTT’,“N/A”},[{variable_map,#{<<“client_id”>> => <<“AckMrCeAAx2bHcmZcnfOzg”>>}},{channels,1},{channel_max,1},{frame_max,0},{client_properties,[{<<“product”>>,longstr,<<“MQTT client”>>},{clie
nt_id,longstr,<<“AckMrCeAAx2bHcmZcnfOzg”>>}]},{ssl,false}]},none,<0.530.0>,{auth_state,<<“chirpstack_as”>>,{user,<<“chirpstack_as”>>,[‘Chirpstack’],[{rabbit_auth_backend_internal,none}]},<<“/”>>},#Fun<rabbit_mqtt_processor.0.50925783>,{0
,0,0,0,0,65535,2653,1},#Fun<rabbit_mqtt_util.4.100928214>,#Fun<rabbit_mqtt_util.5.100928214>},<0.32622.304>,{state,fine,5000,#Ref<0.2613465786.3301179395.25474>}}
** Reason for termination ==
** {{badmatch,{timeout,{mqtt_node,‘rabbit@loraapp-srv-01’}}},[{rabbit_mqtt_processor,close_connection,1,[{file,“src/rabbit_mqtt_processor.erl”},{line,900}]},{rabbit_mqtt_reader,do_terminate,2,[{file,“src/rabbit_mqtt_reader.erl”},{line,24
2}]},{gen_server2,terminate,3,[{file,“src/gen_server2.erl”},{line,1169}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,259}]}]}
** In ‘terminate’ callback with reason ==
** {shutdown,conn_closed}
2021-06-03 04:01:36.291 [error] <0.32645.304> CRASH REPORT Process <0.32645.304> with 1 neighbours exited with reason: no match of right hand value {timeout,{mqtt_node,‘rabbit@loraapp-srv-01’}} in rabbit_mqtt_processor:close_connection/1
line 900 in gen_server2:terminate/3 line 1172
2021-06-03 04:01:36.292 [error] <0.32621.304> Supervisor {<0.32621.304>,amqp_channel_sup} had child channel started with amqp_channel:start_link(direct, <0.32622.304>, 1, <0.32619.304>, {<<“10.93.0.1:44382 → 10.93.0.2:1883”>>,1}) at <0.
32618.304> exit with reason no match of right hand value {timeout,{mqtt_node,‘rabbit@loraapp-srv-01’}} in rabbit_mqtt_processor:close_connection/1 line 900 in context child_terminated
2021-06-03 04:01:36.292 [error] <0.32640.304> Supervisor {<0.32640.304>,rabbit_mqtt_connection_sup} had child rabbit_mqtt_reader started with rabbit_mqtt_reader:start_link(<0.32638.304>, {acceptor,{0,0,0,0,0,0,0,0},1883}) at <0.32645.304exit with reason no match of right hand value {timeout,{mqtt_node,‘rabbit@loraapp-srv-01’}} in rabbit_mqtt_processor:close_connection/1 line 900 in context child_terminated
2021-06-03 04:01:36.292 [error] <0.32621.304> Supervisor {<0.32621.304>,amqp_channel_sup} had child channel started with amqp_channel:start_link(direct, <0.32622.304>, 1, <0.32619.304>, {<<“10.93.0.1:44382 → 10.93.0.2:1883”>>,1}) at <0.
32618.304> exit with reason reached_max_restart_intensity in context shutdown
2021-06-03 04:01:36.292 [error] <0.32640.304> Supervisor {<0.32640.304>,rabbit_mqtt_connection_sup} had child rabbit_mqtt_reader started with rabbit_mqtt_reader:start_link(<0.32638.304>, {acceptor,{0,0,0,0,0,0,0,0},1883}) at <0.32645.304
exit with reason reached_max_restart_intensity in context shutdown
2021-06-03 04:01:36.292 [error] <0.32622.304> ** Generic server <0.32622.304> terminating
** Last message in was {‘EXIT’,<0.32645.304>,{{badmatch,{timeout,{mqtt_node,‘rabbit@loraapp-srv-01’}}},[{rabbit_mqtt_processor,close_connection,1,[{file,“src/rabbit_mqtt_processor.erl”},{line,900}]},{rabbit_mqtt_reader,do_terminate,2,[{f
ile,“src/rabbit_mqtt_reader.erl”},{line,242}]},{gen_server2,terminate,3,[{file,“src/gen_server2.erl”},{line,1169}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,259}]}]}}
** When Server state == {state,amqp_direct_connection,{state,‘rabbit@loraapp-srv-01’,{user,<<“chirpstack_as”>>,[‘Chirpstack’],[{rabbit_auth_backend_internal,none}]},<<“/”>>,{amqp_params_direct,<<“chirpstack_as”>>,<<“SUhIVgLkkrUfG6R/boaoY
Fhn9qAYI+ygy5qch5P7abl3ORsRMBTw4JnorNUjeMKW”>>,<<“/”>>,‘rabbit@loraapp-srv-01’,{amqp_adapter_info,{0,0,0,0,0,65535,2653,2},1883,{0,0,0,0,0,65535,2653,1},44382,<<“10.93.0.1:44382 → 10.93.0.2:1883”>>,{‘MQTT’,“3.1.1”},[{variable_map,#{<<“c
lient_id”>> => <<“AckMrCeAAx2bHcmZcnfOzg”>>}},{channels,1},{channel_max,1},{frame_max,0},{client_properties,[{<<“product”>>,longstr,<<“MQTT client”>>},{client_id,longstr,<<“AckMrCeAAx2bHcmZcnfOzg”>>}]},{ssl,false}]},},{amqp_adapter_inf
o,{0,0,0,0,0,65535,2653,2},1883,{0,0,0,0,0,65535,2653,1},44382,<<“10.93.0.1:44382 → 10.93.0.2:1883”>>,{‘MQTT’,“3.1.1”},[{variable_map,#{<<“client_id”>> => <<“AckMrCeAAx2bHcmZcnfOzg”>>}},{channels,1},{channel_max,1},{frame_max,0},{client
_properties,[{<<“product”>>,longstr,<<“MQTT client”>>},{client_id,longstr,<<“AckMrCeAAx2bHcmZcnfOzg”>>}]},{ssl,false}]},<0.32624.304>,undefined,1621256199087},<0.32637.304>,{amqp_params_direct,<<“chirpstack_as”>>,<<“SUhIVgLkkrUfG6R/boaoY
Fhn9qAYI+ygy5qch5P7abl3ORsRMBTw4JnorNUjeMKW”>>,<<“/”>>,‘rabbit@loraapp-srv-01’,{amqp_adapter_info,{0,0,0,0,0,65535,2653,2},1883,{0,0,0,0,0,65535,2653,1},44382,<<“10.93.0.1:44382 → 10.93.0.2:1883”>>,{‘MQTT’,“3.1.1”},[{variable_map,#{<<“c
lient_id”>> => <<“AckMrCeAAx2bHcmZcnfOzg”>>}},{channels,1},{channel_max,1},{frame_max,0},{client_properties,[{<<“product”>>,longstr,<<“MQTT client”>>},{client_id,longstr,<<“AckMrCeAAx2bHcmZcnfOzg”>>}]},{ssl,false}]},},0,[{<<“capabilities”>>,table,[{<<“publisher_confirms”>>,bool,true},{<<“exchange_exchange_bindings”>>,bool,true},…]},…],…}
** Reason for termination ==
** “stopping because dependent process <0.32645.304> died: {{badmatch,\n {timeout,\n {mqtt_node,\n ‘rabbit@loraapp-srv-01’}}},\n [{rabbit_mqtt_processor,\n close_connection,1,\n [{file,\n "src/rabbit_mqtt_processor.erl"},\n {line,900}]},\n {rabbit_mqtt_reader,\n do_terminate,2,\n [{file,\n "src/rabbit_mqtt_reader.erl"},\n {line,242}]},\n {gen_server2,\n terminate,3,\n [{file,\n "src/gen_server2.erl"},\n {line,1169}]},\n {proc_lib,wake_up,3,\n [{file,\n "proc_lib.erl"},\n {line,25…”
2021-06-03 04:01:36.293 [error] <0.32622.304> CRASH REPORT Process <0.32622.304> with 0 neighbours exited with reason: “stopping because dependent process <0.32645.304> died: {{badmatch,\n {timeout,\n {mqtt_node,\n ‘rabbit@loraapp-srv-01’}}},\n [{rabbit_mqtt_processor,\n close_connection,1,\n …” in gen_server:handle_common_reply/8 line 751
2021-06-03 04:01:36.293 [error] <0.32641.304> Supervisor {<0.32641.304>,amqp_connection_sup} had child connection started with amqp_gen_connection:start_link(<0.32651.304>, {amqp_params_direct,<<“chirpstack_as”>>,<<“SUhIVgLkkrUfG6R/boaoYFhn9qAYI+ygy5qch5P7abl3ORsRMBTw4Jn…”>>,…}) at <0.32622.304> exit with reason “stopping because dependent process <0.32645.304> died: {{badmatch,\n {timeout,\n {mqtt_node,\n ‘rabbit@loraapp-srv-01’}}},\n [{rabbit_mqtt_processor,\n close_connection,1,\n …” in context child_terminated
2021-06-03 04:01:36.293 [error] <0.32641.304> Supervisor {<0.32641.304>,amqp_connection_sup} had child connection started with amqp_gen_connection:start_link(<0.32651.304>, {amqp_params_direct,<<“chirpstack_as”>>,<<“SUhIVgLkkrUfG6R/boaoYFhn9qAYI+ygy5qch5P7abl3ORsRMBTw4Jn…”>>,…}) at <0.32622.304> exit with reason reached_max_restart_intensity in context shutdown
2021-06-03 04:01:36.293 [error] <0.295.305> ** Generic server <0.295.305> terminating
** Last message in was {‘EXIT’,<0.32618.304>,{{badmatch,{timeout,{mqtt_node,‘rabbit@loraapp-srv-01’}}},[{rabbit_mqtt_processor,close_connection,1,[{file,“src/rabbit_mqtt_processor.erl”},{line,900}]},{rabbit_mqtt_reader,do_terminate,2,[{file,“src/rabbit_mqtt_reader.erl”},{line,242}]},{gen_server2,terminate,3,[{file,“src/gen_server2.erl”},{line,1169}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,259}]}]}}
** When Server state == {ch,{conf,running,rabbit_framing_amqp_0_9_1,1,<0.32618.304>,<0.32618.304>,<0.32622.304>,<<“10.93.0.1:44382 → 10.93.0.2:1883”>>,undefined,{user,<<“chirpstack_as”>>,[‘Chirpstack’],[{rabbit_auth_backend_internal,none}]},<<“/”>>,<<“mqtt-subscription-AckMrCeAAx2bHcmZcnfOzgqos0”>>,<0.32624.304>,[{<<“publisher_confirms”>>,bool,true},{<<“exchange_exchange_bindings”>>,bool,true},{<<“basic.nack”>>,bool,true},{<<“consumer_cancel_notify”>>,bool,true},{<<“connection.blocked”>>,bool,true},{<<“authentication_failure_close”>>,bool,true}],none,10,134217728,undefined,#{<<“client_id”>> => <<“AckMrCeAAx2bHcmZcnfOzg”>>}},{lstate,<0.289.305>,false},none,1,{0,{,}},#{<0.458.0> => {resource,<<“/”>>,queue,<<“mqtt-subscription-fe80:1:3:26002:1:1:1:27qos1”>>},<0.24213.212> => {resource,<<“/”>>,queue,<<“mqtt-subscription-logger-2755qos0”>>},<0.24242.212> => {resource,<<“/”>>,queue,<<“mqtt-subscription-logger-1422qos0”>>},<0.32291.304> => {resource,<<“/”>>,queue,<<“mqtt-subscription-AckMrCeAAx2bHcmZcnfOzgqos0”>>},<0.6870.353> => {resource,<<“/”>>,queue,<<“mqtt-subscription-mqtt_15e1c9e9.577696qos1”>>}},{state,#{<0.458.0> => #Ref<0.2613465786.3254779908.20375>,<0.24213.212> => #Ref<0.2613465786.3479699457.204624>,<0.24242.212> => #Ref<0.2613465786.3479699457.204623>,<0.32291.304> => #Ref<0.2613465786.3479699457.202162>,<0.6870.353> => #Ref<0.2613465786.3520331777.196060>},erlang},#{<<“amq.ctag-YOowwvGDd4Gio7wHkLEpDQ”>> => {{amqqueue,{resource,<<“/”>>,queue,<<“mqtt-subscription-AckMrCeAAx2bHcmZcnfOzgqos0”>>},false,true,none,,<0.32291.304>,,,,undefined,undefined,,,live,0,,<<“/”>>,#{user => <<“chirpstack_as”>>},rabbit_classic_queue,#{}},{true,10,false,}}},#{<0.32291.304> => {1,{<<“amq.ctag-YOowwvGDd4Gio7wHkLEpDQ”>>,…}}},…}
** Reason for termination ==
** {{badmatch,{timeout,{mqtt_node,‘rabbit@loraapp-srv-01’}}},[{rabbit_mqtt_processor,close_connection,1,[{file,“src/rabbit_mqtt_processor.erl”},{line,900}]},{rabbit_mqtt_reader,do_terminate,2,[{file,“src/rabbit_mqtt_reader.erl”},{line,242}]},{gen_server2,terminate,3,[{file,“src/gen_server2.erl”},{line,1169}]},{proc_lib,wake_up,3,[{file,“proc_lib.erl”},{line,259}]}]}
2021-06-03 04:01:36.294 [error] <0.295.305> CRASH REPORT Process <0.295.305> with 0 neighbours exited with reason: no match of right hand value {timeout,{mqtt_node,‘rabbit@loraapp-srv-01’}} in rabbit_mqtt_processor:close_connection/1 line 900 in gen_server2:terminate/3 line 1183
2021-06-03 04:01:36.294 [error] <0.292.305> Supervisor {<0.292.305>,rabbit_channel_sup} had child channel started with rabbit_channel:start_link(1, <0.32618.304>, <0.32618.304>, <0.32622.304>, <<“10.93.0.1:44382 → 10.93.0.2:1883”>>, rabbit_framing_amqp_0_9_1, {user,<<“chirpstack_as”>>,[‘Chirpstack’],[{rabbit_auth_backend_internal,none}]}, <<“/”>>, [{<<“publisher_confirms”>>,bool,true},{<<“exchange_exchange_bindings”>>,bool,true},{<<“basic.nack”>>,…},…], <0.32624.304>, <0.289.305>, {amqp_params_direct,<<“chirpstack_as”>>,<<“SUhIVgLkkrUfG6R/boaoYFhn9qAYI+ygy5qch5P7abl3ORsRMBTw4Jn…”>>,…}) at <0.295.305> exit with reason no match of right hand value {timeout,{mqtt_node,‘rabbit@loraapp-srv-01’}} in rabbit_mqtt_processor:close_connection/1 line 900 in context child_terminated
2021-06-03 04:01:36.294 [error] <0.292.305> Supervisor {<0.292.305>,rabbit_channel_sup} had child channel started with rabbit_channel:start_link(1, <0.32618.304>, <0.32618.304>, <0.32622.304>, <<“10.93.0.1:44382 → 10.93.0.2:1883”>>, rabbit_framing_amqp_0_9_1, {user,<<“chirpstack_as”>>,[‘Chirpstack’],[{rabbit_auth_backend_internal,none}]}, <<“/”>>, [{<<“publisher_confirms”>>,bool,true},{<<“exchange_exchange_bindings”>>,bool,true},{<<“basic.nack”>>,…},…], <0.32624.304>, <0.289.305>, {amqp_params_direct,<<“chirpstack_as”>>,<<“SUhIVgLkkrUfG6R/boaoYFhn9qAYI+ygy5qch5P7abl3ORsRMBTw4Jn…”>>,…}) at <0.295.305> exit with reason reached_max_restart_intensity in context shutdown
2021-06-03 04:01:37.105 [info] <0.24828.366> MQTT vhost picked using plugin configuration or default
2021-06-03 04:01:42.113 [error] <0.24828.366> MQTT cannot accept a connection: client ID registration timed out
2021-06-03 04:01:42.113 [error] <0.24828.366> MQTT cannot accept connection 10.93.0.1:35338 → 10.93.0.2:1883 due to an internal error or unavailable component
2021-06-03 04:01:46.115 [info] <0.25564.366> MQTT vhost picked using plugin configuration or default
2021-06-03 04:01:46.662 [info] <0.25564.366> accepting MQTT connection <0.25564.366> (10.93.0.1:35358 → 10.93.0.2:1883, client id: 1M4VrmghPWJsSOFRncBEeA)
2021-06-04 03:07:30.005 [info] <0.26138.369> accepting AMQP connection <0.26138.369> (10.104.1.131:46242 → 10.93.0.2:5672)
{bad_header,<<“GET / HT”>>}
2021-06-04 03:07:35.040 [error] <0.26349.369> MQTT cannot parse a frame on connection ‘10.104.1.131:60008 → 10.93.0.2:1883’, unparseable payload: <<73,0,0,0,102,0,0,0>>, error: {{badmatch,<<>>}, [{rabbit_mqtt_frame,parse_frame,3,[{file,“src/rabbit_mqtt_frame.erl”},{line,104}]},{rabbit_mqtt_reader,parse,2,[{file,“src/rabbit_mqtt_reader.erl”},{line,358}]},{rabbit_mqtt_reader,process_received_bytes,2,[{file,“src/rabbit_mqtt_reader.erl”},{line,289}]},{gen_server2,handle_msg,2,[{file,“src/gen_server2.erl”},{line,1067}]},{proc_lib,init_p_do_apply,3,[{file,“proc_lib.erl”},{line,249}]}]}
2021-06-04 03:07:35.047 [info] <0.24610.369> accepting AMQP connection <0.24610.369> (10.104.1.131:48490 → 10.93.0.2:5672)
2021-06-04 03:07:35.047 [error] <0.24610.369> closing AMQP connection <0.24610.369> (10.104.1.131:48490 → 10.93.0.2:5672):
{bad_header,<<0,0,0,22,128,1,0,1>>}
2021-06-04 03:07:35.054 [info] <0.26587.369> accepting AMQP connection <0.26587.369> (10.104.1.131:48500 → 10.93.0.2:5672)
2021-06-04 03:07:35.054 [error] <0.26587.369> closing AMQP connection <0.26587.369> (10.104.1.131:48500 → 10.93.0.2:5672):
{bad_version,{0,0,0,0}}
2021-06-04 03:07:35.063 [info] <0.26586.369> accepting AMQP connection <0.26586.369> (10.104.1.131:48506 → 10.93.0.2:5672)
2021-06-04 03:07:35.068 [info] <0.26439.369> accepting AMQP connection <0.26439.369> (10.104.1.131:48512 → 10.93.0.2:5672)
2021-06-04 03:07:35.068 [error] <0.26439.369> closing AMQP connection <0.26439.369> (10.104.1.131:48512 → 10.93.0.2:5672):
{bad_version,{2,0,9,1}}
2021-06-04 03:07:36.067 [warning] <0.26586.369> closing AMQP connection <0.26586.369> (10.104.1.131:48506 → 10.93.0.2:5672):
client unexpectedly closed TCP connection
2021-06-04 03:08:22.338 [error] <0.28654.369> MQTT cannot parse a frame on connection ‘10.104.1.131:51810 → 10.93.0.2:1883’, unparseable payload: <<“GET / HTTP/1.1\r\nConnection: Close\r\nHost: loraapp-srv-01\r\nPragma: no-cache\r\nUser-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)\r\nAccept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, /\r\nAccept-Language: en\r\nAccept-Charset: iso-8859-1,*,utf-8\r\n\r\n”>>, error: {{badmatch,<<“T / HTTP/1.1\r\nConnection: Close\r\nHost: loraapp-srv-01\r\nPragma: no-cac”>>}, [{rabbit_mqtt_frame,parse_frame,3,[{file,“src/rabbit_mqtt_frame.erl”},{line,104}]},{rabbit_mqtt_reader,parse,2,[{file,“src/rabbit_mqtt_reader.erl”},{line,358}]},{rabbit_mqtt_reader,process_received_bytes,2,[{file,“src/rabbit_mqtt_reader.erl”},{line,289}]},{gen_server2,handle_msg,2,[{file,“src/gen_server2.erl”},{line,1067}]},{proc_lib,init_p_do_apply,3,[{file,“proc_lib.erl”},{line,249}]}]}