Regular Flow: Deployment: Flow is stuck in 'In_Progress' status

[yuliiamir]

The issue occurred during JJ execution and a lot of flows were stuck in 'In Progress' state. Unfortunately, all logs from Kilda were not collected, but the main logs contain the following errors:

2024-02-20T11:51:18.591 Flow creation log:

{
      "_index" : "kilda-2024.02.20",
      "_type" : "kilda-storm",
      "_id" : "AY3GWyVPNCoVO_HYZ_B8",
      "_score" : 1.0,
      "_source" : {
        "loggerFqcn" : "org.apache.logging.slf4j.Log4jLogger",
        "level" : "INFO",
        "endOfBatch" : false,
        "thread" : "Thread-23-FLOW_CREATE_HUB-executor[1 1]",
        "message" : "Create the flow: 20Feb115118_384_tahini5774, source 00:00:00:00:00:00:00:01_10_4036, destination 00:00:00:00:00:00:00:02_10_3364, diverse flowId null, bandwidth 500, path computation strategy COST_AND_AVAILABLE_BANDWIDTH, max latency null, max latency tier2 null",
        "threadPriority" : 5,
        "type" : "kilda-storm",
        "tags" : [ "storm-worker_log" ],
        "threadId" : 60,
        "@timestamp" : "2024-02-20T11:51:18.591Z",
        "port" : 56218,
        "@version" : "1",
        "host" : "172.18.0.25",
        "loggerName" : "org.openkilda.wfm.topology.flowhs.fsm.create.FlowCreateFsm",
        "contextMap" : {
          "event_type" : "flow_create",
          "kafka.partition" : "kilda.northbound.flowhs.priv-0",
          "flow_id" : "20Feb115118_384_tahini5774",
          "kafka.offset" : "2",
          "FLOW_OPERATIONS_DASHBOARD" : "flow-create",
          "correlation_id" : "1060151b-3304-4c3a-a410-245a87463723 : fn-tests-3a20fcb6-e938-4420-8833-43a8b563188b",
          "topology_name" : "flowhs_blue"
        },
        "timeMillis" : 1708429878589
      }
    }

2024-02-20T11:51:19.530 Rule installation

{
      "_index" : "kilda-2024.02.20",
      "_type" : "kilda-FL",
      "_id" : "AY3GWyjyNCoVO_HYZ_GL",
      "_score" : 1.0,
      "_source" : {
        "role" : "mgmt",
        "level" : "INFO",
        "thread" : "pool-5-thread-4",
        "message" : "Command <egress-flow-segment-INSTALL{id=5c58439f-cfe6-11ee-a773-ed1543c2577a, metadata=FlowSegmentMetadata(flowId=20Feb115118_384_tahini5774, cookie=0x400000000000F16B, multiTable=true), endpoint=switchId=\"00:00:00:00:00:00:00:02\" port=10 vlanId=3364, ingressEndpoint=switchId=\"00:00:00:00:00:00:00:01\" port=10 vlanId=4036, isl_port=4, encapsulation=FlowTransitEncapsulation(id=2604, type=TRANSIT_VLAN)}> on 00:00:00:00:00:00:00:02 completed with report <FlowSegmentReport{commandId=5c58439f-cfe6-11ee-a773-ed1543c2577a, success}> (exec time: PT0.002S)",
        "type" : "kilda-FL",
        "tags" : [ "kilda-floodlight" ],
        "@timestamp" : "2024-02-20T11:51:19.530Z",
        "port" : 36836,
        "@version" : 1,
        "host" : "172.18.0.21",
        "loggerName" : "org.openkilda.floodlight.command.SpeakerCommandProcessor",
        "contextMap" : {
          "correlation_id" : "5c570b1d-cfe6-11ee-a773-cf9cd77bfb50 : 1060151b-3304-4c3a-a410-245a87463723 : fn-tests-3a20fcb6-e938-4420-8833-43a8b563188b"
        },
        "region" : "2",
        "decoded_latency_ms" : null
      }
    }

2024-02-20T11:51:29.635 Timeout log:

{
      "_index" : "kilda-2024.02.20",
      "_type" : "kilda-storm",
      "_id" : "AY3GW1BrNCoVO_HYZ_vy",
      "_score" : 1.0,
      "_source" : {
        "loggerFqcn" : "org.apache.logging.slf4j.Log4jLogger",
        "level" : "WARN",
        "endOfBatch" : false,
        "thread" : "Thread-31-SPEAKER_WORKER-executor[65 65]",
        "message" : "Timeout occurred while waiting for a response for 5c58439f-cfe6-11ee-a773-ed1543c2577a : 1060151b-3304-4c3a-a410-245a87463723 : fn-tests-3a20fcb6-e938-4420-8833-43a8b563188b",
        "threadPriority" : 5,
        "type" : "kilda-storm",
        "tags" : [ "storm-worker_log" ],
        "threadId" : 68,
        "@timestamp" : "2024-02-20T11:51:29.635Z",
        "port" : 56218,
        "@version" : "1",
        "host" : "172.18.0.25",
        "loggerName" : "org.openkilda.wfm.topology.flowhs.bolts.SpeakerWorkerBolt",
        "contextMap" : {
          "correlation_id" : "1060151b-3304-4c3a-a410-245a87463723 : fn-tests-3a20fcb6-e938-4420-8833-43a8b563188b",
          "kafka.partition" : "kilda.northbound.flowhs.priv-0",
          "topology_name" : "flowhs_blue",
          "kafka.offset" : "2"
        },
        "timeMillis" : 1708429889634
      }
    }

Strict_bandwidth_false__Flow_is_rerouted_even_if_there_is_no_available_bandwidth_on_alternative_path__sets_status_to_Degraded.server.log.json