{"id":1670,"date":"2025-11-03T20:36:13","date_gmt":"2025-11-03T18:36:13","guid":{"rendered":"https:\/\/www.mcgill.org.za\/stuff\/?p=1670"},"modified":"2025-11-04T08:21:52","modified_gmt":"2025-11-04T06:21:52","slug":"frr-says-finite-state-machine-error-for-ipv6-bgp-on-linux","status":"publish","type":"post","link":"https:\/\/www.mcgill.org.za\/stuff\/archives\/1670","title":{"rendered":"FRR says “Finite State Machine Error” for IPv6 BGP on Linux"},"content":{"rendered":"\n

If you turn off link-local addresses for the interface (those horrible 169.254.50.231 type addresses, and also the IPv6 fe80::dead:beef:4:f00d\/64<\/code> things), FRR goes sulky. In netplan, it looks like this:<\/p>\n\n\n

\n
\"FSM\"<\/a>
The official mascot of the BGP Finite State Machine (FSM)<\/figcaption><\/figure>\n<\/div>\n\n\n
ethernets:\n  eth0:\n    link-local: [ ]<\/strong><\/code><\/pre>\n\n\n\n
When FRR is sulky, it says it will advertise IPv6 subnets, but its doesn’t actually advertise the routes, because it doesn’t actually connect to its peer(s).  Here says it will advertise, but it doesn’t:<\/p>\n\n\n\n
# show ip bgp all\n...\nNetwork Next Hop Metric LocPrf Weight Path\n*> 2000:deaf:7012:feed::\/64 :: 0 32768 i<\/code><\/pre>\n\n\n\n
tcpdump<\/code> says this, which doesn’t help much:<\/p>\n\n\n\n
# tcpdump -i any port bgp\n17:27:49.667454 eth0  Out ifindex 3 52:54:00:a1:34:67 ethertype IPv6 (0x86dd), length 113: (class 0xc0, flowlabel 0xdde41, hlim 1, next-header TCP (6) payload length: 53) 2000:deaf::1194.45245 > 2000:deaf::80.179: Flags [P.], cksum 0x445a (incorrect -> 0xdc08), seq 1:22, ack 1, win 507, options [nop,nop,TS val 3099884752 ecr 3196106109], length 21: BGP\n\tNotification Message (3), length: 21, Finite State Machine Error (5) subcode Unspecified Error (0)<\/strong><\/code><\/pre>\n\n\n\n
To turn on debug, it’s a few easy and obvious commands:<\/p>\n\n\n\n
# vtysh\nconfig term\ndebug bgp as4\ndebug bgp bfd\ndebug bgp flowspec\ndebug bgp keepalives\ndebug bgp neighbor-events\ndebug bgp pbr\ndebug bgp updates\ndebug bgp vpn\ndebug bgp bestpath\ndebug bgp evpn\ndebug bgp graceful-restart\ndebug bgp labelpool\ndebug bgp nht\ndebug bgp update-groups\ndebug bgp vnc\ndebug bgp zebra\nlog syslog debug\nend\nterminal monitor<\/code><\/pre>\n\n\n\n
And then the terminal displays all the debug you could want, including the confession below that FRR’s BGP on IPv6 cannot function without a link-local address.  I suppose some IPv6 routers don’t like routing without a Link-Local to talk to … or it’s in the standard, or something stupid:<\/p>\n\n\n\n
.336 [DEBG] bgpd: [YTARA-Q9ZD1] [Event] BGP connection from host 2000:deaf::80 fd 27\n.336 [DEBG] bgpd: [JYX7T-SMTAQ] bgp_peer_gr_init called ..\n.336 [DEBG] bgpd: [YB7SD-E2DZS] [BGP_GR] Peer state changed  --to-->  : 4 : !\n.336 [DEBG] bgpd: [VTCGN-KEKBQ] bgp_peer_gr_flags_update [BGP_GR] called !\n.336 [DEBG] bgpd: [VBM1Z-TD8QM] [BGP_GR] Peer 2000:deaf::1 Flag PEER_FLAG_GRACEFUL_RESTART_HELPER : Set : !\n.336 [DEBG] bgpd: [M49N5-G8MHD] [BGP_GR] Peer 2000:deaf::1 Flag PEER_FLAG_GRACEFUL_RESTART : UnSet : !\n.336 [DEBG] bgpd: [SRJ2F-0FBJK] [BGP_GR] Peer 2000:deaf::1 Flag PEER_FLAG_GRACEFUL_RESTART_GLOBAL_INHERIT : Set : !\n.336 [DEBG] bgpd: [VTCGN-KEKBQ] bgp_peer_gr_flags_update [BGP_GR] called !\n.336 [DEBG] bgpd: [VBM1Z-TD8QM] [BGP_GR] Peer 2000:deaf::1 Flag PEER_FLAG_GRACEFUL_RESTART_HELPER : Set : !\n.336 [DEBG] bgpd: [M49N5-G8MHD] [BGP_GR] Peer 2000:deaf::1 Flag PEER_FLAG_GRACEFUL_RESTART : UnSet : !\n.336 [DEBG] bgpd: [SRJ2F-0FBJK] [BGP_GR] Peer 2000:deaf::1 Flag PEER_FLAG_GRACEFUL_RESTART_GLOBAL_INHERIT : Set : !\n.336 [DEBG] bgpd: [WNKP5-SN018] Found existing bnc 2000:deaf::1\/128(0)(VRF default) flags 0xf ifindex 0 #paths 0 peer 0x7ebfd9c26031\n.336 [DEBG] bgpd: [T91AW-FGMHW] bgp_fsm_change_status : vrf default(0), Status: Active established_peers 0\n.336 [DEBG] bgpd: [ZQHFG-DQGX1] 2000:deaf::1 went from Idle to Active\n.337 [DEBG] bgpd: [ZWCSR-M7FG9] 2000:deaf::1 [FSM] TCP_connection_open (Active->OpenSent), fd 27\n.337 [WARN] bgpd: [ZM2F8-MV4BJ][EC 33554509] Interface: eth0 does not have a v6 LL address associated with it, waiting until one is created for it<\/strong>\n.337 [ERR!] bgpd: [M3MYP-BVWDS][EC 33554460] 2000:deaf::1: nexthop_set failed, resetting connection - intf eth0\n.337 [ERR!] bgpd: [NQGZV-Y3W62][EC 100663299] bgp_connect_success: bgp_getsockname(): failed for peer 2000:deaf::1, fd 27\n.337 [INFO] bgpd: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 2000:deaf::1 5\/0 (Neighbor Events Error\/Unspecific) 0 bytes\n.337 [DEBG] bgpd: [VTCGN-KEKBQ] bgp_peer_gr_flags_update [BGP_GR] called !\n.337 [DEBG] bgpd: [VBM1Z-TD8QM] [BGP_GR] Peer 2000:deaf::1 Flag PEER_FLAG_GRACEFUL_RESTART_HELPER : Set : !\n.337 [DEBG] bgpd: [M49N5-G8MHD] [BGP_GR] Peer 2000:deaf::1 Flag PEER_FLAG_GRACEFUL_RESTART : UnSet : !\n.337 [DEBG] bgpd: [SRJ2F-0FBJK] [BGP_GR] Peer 2000:deaf::1 Flag PEER_FLAG_GRACEFUL_RESTART_GLOBAL_INHERIT : Set : !\n.337 [DEBG] bgpd: [ZWCSR-M7FG9] 2000:deaf::1 [FSM] BGP_Stop (Active->Idle), fd 27\n.339 [DEBG] bgpd: [T91AW-FGMHW] bgp_fsm_change_status : vrf default(0), Status: Deleted established_peers 0\n.339 [DEBG] bgpd: [ZQHFG-DQGX1] 2000:deaf::1 went from Active to Deleted\n\nTo turn on link-local for \n<\/code><\/pre>\n\n\n\n
And the problem is fixed by turning on the (rather default) link-local address for IPv6.  In netplan, it’s this part:<\/p>\n\n\n\n
ethernets:\n  eth0:\n    link-local: [ ipv6 ]<\/strong><\/code><\/pre>\n\n\n\n
And then<\/em> it works.<\/p>\n","protected":false},"excerpt":{"rendered":"
If you turn off link-local addresses for the interface (those horrible 169.254.50.231 type addresses, and also the IPv6 fe80::dead:beef:4:f00d\/64 things), FRR goes sulky. In netplan, it looks like this: When FRR is sulky, it says it will advertise IPv6 subnets, … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":1672,"comment_status":"open","ping_status":null,"sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[341,342,310,33,190,11],"class_list":["post-1670","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-stuff","tag-bgp","tag-frr","tag-ipv6","tag-networking","tag-stuff","tag-stupidity"],"_links":{"self":[{"href":"https:\/\/www.mcgill.org.za\/stuff\/wp-json\/wp\/v2\/posts\/1670","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.mcgill.org.za\/stuff\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.mcgill.org.za\/stuff\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.mcgill.org.za\/stuff\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.mcgill.org.za\/stuff\/wp-json\/wp\/v2\/comments?post=1670"}],"version-history":[{"count":6,"href":"https:\/\/www.mcgill.org.za\/stuff\/wp-json\/wp\/v2\/posts\/1670\/revisions"}],"predecessor-version":[{"id":1677,"href":"https:\/\/www.mcgill.org.za\/stuff\/wp-json\/wp\/v2\/posts\/1670\/revisions\/1677"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.mcgill.org.za\/stuff\/wp-json\/wp\/v2\/media\/1672"}],"wp:attachment":[{"href":"https:\/\/www.mcgill.org.za\/stuff\/wp-json\/wp\/v2\/media?parent=1670"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.mcgill.org.za\/stuff\/wp-json\/wp\/v2\/categories?post=1670"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.mcgill.org.za\/stuff\/wp-json\/wp\/v2\/tags?post=1670"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}