Max flows parameter can be exceeded for a given slice

[gth828r]

I was playing around with Floodlight static flow pusher on top of FSFW, and I was able to create 18 flows in an ovs instance when my slice limit was set to 10:

{
    "DPID": "d0:0d:de:ad:be:a7:da:d5", 
    "connected": true, 
    "flow_rate": 1111.1111111111111, 
    "max_flow_rule": 1, 
    "max_flows": 10, 
    "max_packet_in_rate": 1, 
    "packet_in_rate": 0.016272031924424873, 
    "total_flows": 18
}

I sent flows that did the following:

• Match VLAN ID 100
• Match MAC address (incrementing the src MAC by 1 for each flow)
• Output to a specific port (2)

My slice included ports 1 and 2, and VLAN 100 on both ports. By not specifying the in_port, I believe that FSFW split my flowmods into two flowmods, which match on in_port 1 or in_port 2. I have not investigated what caused the 18 flows (flow accounting issues vs. race conditions, etc).

Here is the resulting flowtable as reported by Open vSwitch:
https://gist.github.com/gth828r/e2efcdea6e87d647c6b1#file-gistfile1-txt

[ajragusa]

Hi Tim

How many flows did you try and push? Did you have some logs from FSFW when you were doing this? I would like to see the logs of possible.

Also what version of FSFW were you running 1.0.4b?

Thanks

[gth828r]

Yes to 1.0.4b, and yes I do have the logs. I unfortunately don't know exactly when the problem happened, because I didn't notice the problem at first, so my apologies for that. Here is a ~2000 line gist of the logs, from startup until a time where I know the problem should have already happened by:
https://gist.github.com/gth828r/1ddb13ed89ba57de8015

I suspect the problem occurred toward the latter half of the logs. I am working within the "samevlan" slice, and the "diffvlan" slice intentionally doesn't have a controller for the duration of this test.

Here is my fsfw config file as well, in case it is useful.
https://gist.github.com/gth828r/126827c92cd3c34a2bc0

This was one stress test against the various tests against fsfw in a series of them. If you want to know more about the steps leading up to this test, I can describe it. FYI, I have not been able to reproduce this again, but I'll be trying many times, and I'll report back if I see it again (hopefully with a more narrow log snippet!)

[gth828r]

Oh, and in terms of the number of flows I was trying to push, I think I was trying to push 10 (without the in_port specified in the match). Note that SFP had previously also had one flow installed prior to this test, but that test was verifying that the slice was disable as part of violating the packet-in rate. I then re-enabled the slice, but I am assuming those flows got removed when the slice was disabled.

[gth828r]

I have not been able to replicate this at all in my testing. Perhaps I was just doing something funky in my initial exploration, but I am not sure what that would have been.

Anyway, feel free to do as you wish with this ticket. I don't have a good way to replicate the issue, so I won't hold it against you all if you close it. :)

[ajragusa]

When we start testing the next release of FSFW we will attempt to reproduce what you described. If we can't replicate then we will close.

[ajragusa]

I think I figured out how to replicate what you saw, however I'm not sure there is much we can do about it.

Here is the situation

You add 10 flows to the switch, the switch disconnects and reconnects (at this point fsfw says you have 0 flows installed) then you attempt to push 10 more flows. Because FSFW says you have 0 flows installed it lets you push them. Once FSFW gets the flow stats it sees you have 20 flows (or 18 in your case) and sets the value to 20. The only option I see here is not letting you send flows until we have received flowstats from the switch and sliced them.

[gth828r]

Ah, I see. Nice find!

In the GENI context, my opinion is that we'd always want to flush rules on a switch reconnect, which would bypass this issue. In general, if the substrate was disconnected, I don't want to make any assumptions about what the switch did while it was disconnected, and since GENI slices aren't really at a high level of production, clearing out the switch's forwarding rules is sufficient.

If you wanted to avoid this problem outside of the GENI context, one other option might be to just freeze the flow stats on a switch disconnect and keep them cached. This assumes that the common case of a switch disconnect is temporary, and that the switch isn't necessarily going to clear out any flows on its own. Once the switch reconnects and fsfw pulls in the latest flow stats, you'd have the correct info, but you wouldn't have as much of a risk of slices going over their flow limit as in the current state.

I'll leave it up to you which approach you think is best.

[ajragusa]

I'm going to take this out of 1.0.5 for now, we will come back to it later

[gth828r]

Sounds good. It seems this problem is not so common in practice, it is probably not catastrophic if it happens, and the right approach requires some thinking.