[stratum-dev] BMv2 standalone instances unable to forward traffic

A Sydney asydney007 at gmail.com
Mon May 3 19:58:04 UTC 2021

I was able to resolve the issue. Below is a quick summary:
Problem: The ONOS GUI did not show any links and further investigation
showed that LLDP traffic was not observed in the stratum_bmv2 log.
Solution: I specified the "cpu_port" argument when stratum_bmv2 starts-up
(i.e. cpu_port was set to 255, which is the same port used in the p4

PS. Feel free to read below for more details:


On Mon, Apr 26, 2021 at 1:41 PM A Sydney <asydney007 at gmail.com> wrote:

> Hi Stratum folks,
>                            It appears that traffic is not forwarded by the
> BMv2 standalone instances. Here are more details:
> 1. I used Max's instructions to create standalone BMv2 instances running
> on Debian 10.
> 2. I created an SdnTest app on ONOS with the identical pipeconf,
> bmv2.json, and p4info.txt files from the ngsdn-tutorial app (i.e. the same
> working versions of these files from the ngsdn-tutorial VM are used in a
> different setup with a different ONOS controller and 6 standalone bmv2
> switches). (https://github.com/opennetworkinglab/ngsdn-tutorial).
> 3. I created a custom 6-node Clos topology and I've attached the
> corresponding netcfg.json in the tarball.
> 4. Onos starts up just fine and I ensure that all apps (similar to the
> ones running on ngsdn-tutorial) are fired-up (See onos_cli_log in attached).
> 5. I then push netcfg.json to onos, and the onos logs show a few "Failed
> to register Bandwidth" and "Failed to register Port" errors (See onos_log
> in attached tarball): Note: After digging a bit, it appears that these
> messages originate from ResourceDeviceListener.java on line 162 when
> "log.isTraceEnabled()" is not set in ConsistentResourceStore on line 122
> (So this essentially may not be an issue?). In any case, onos_cli_log shows
> that all devices are connected and available. Interestingly, the "Tx"
> portstat remains at 0. Given the fact the the lldpprovider app is running,
> I would expect at least LLDP traffic to be flowing.
> 6. As an example, I logged onto leaf1 to take a look at the logs, and from
> the terminal, I see the following (See leaf1_stratum_terminal_log for more
> details):
> """
> E20210426 16:01:18.918056  5458 p4_service.cc:311] Failed to write
> forwarding entries to node 1:

This occurs when an attempt is made to install a flow that already exists.

> """
> So when the switch starts up, the initial "chassis_config_file" lists all
> 8 ports on all switches. However, netcfg.json specifies 6 of the 8. When I
> take a look through the onos web GUI, I still see all 8 ports which implies
> that perhaps there may be issues pushing the updated chassis_config_file to
> the switch?
> 7. When I look through the details debug log for leaf1 (see
> leaf1_log_written_to_file), I only see 1 LLDP pkt (i.e. ethtype 88cc),
> though tcpdump from the controller shows that the controller is indeed
> sending numerous discovery pkts.
> This turns out to be the log when lldphostprovider adds the LLDP rule to
the switch.

> PS. I've attached a tarball with the various log files and a ReadMe
> describing the contents of each file.
> Can you kindly provide feedback on perhaps why traffic (at least LLDP) is
> not forwarded by the switches?

Using tcpdump, I observed that LLDP traffic was sent from the controller
and made it to the switch. However, the bmv2 logs did not register any LLDP
traffic on port 255 (i.e. the CPU port specified in the p4 program). I took
a look at how stratum_bmv2 starts from the ngsdn-tutorial vm and observed
that "cpu_port" was one of the arguments and was set to 255. For some
reason, I thought that 255 was the default CPU port but it turns out that
the default port is 64.  So I specified the "cpu_port" argument in
testrig and pkt-in/outs now work and hence LLDP works.

> BTW. I also tried to connect to leaf1 via "util/p4rt-sh", install flow
> rules to foward traffic between ports 5 and 6, but a quick ping showed that
> traffic was not flowing: I also used tcpdump on leaf1 to verify that indeed
> no pkts were making it on ports 5 and 6 (except dhcp which is expected).
This was a result of the behavior of the protocols in my environment. I had
to add new tables and their corresponding ONOS components, and now traffic
flows just fine.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.stratumproject.org/archives/stratum-dev/attachments/20210503/060cd142/attachment.html>

More information about the stratum-dev mailing list