Applicability of MIMO as Metric for Frame Latency

Arjan Durresi, Raj Jain, Gojko Babic

Department of Computer and Information Science, The Ohio State University,

2015 Neil Ave, Columbus, OH 43210-1277
Phone: 614-688-5610, Fax: 614-292-2911, Email: durresi@cse.wustl.edu

Abstract

In this paper we analyze and compare the use of MIMO versus other metrics and in particular versus LCD for measuring the frame latency in performance testing. The main advantage of using MIMO as frame latency metric is that MIMO is more related to the performance of the switch itself. Also we show that MIMO can be aggregate and that the formulation of MIMO aggregation is very useful in understanding the contribution of each network element and their interconnections in the total frame latency.

Keywords: Frame latency, Performance metrics, Performance testing, QoS parameters, ATM switch performance.

1. Introduction

Frame latency is of particular interest as a QoS parameter, because the frame level is more likely to influence the application latency and performance. Cell-level metrics do not very often reflect the performance as experienced (or desired) by end users. For example, a video user sending 30 frames/sec would like frames to be completely delivered every 33 ms and it does not matter whether the cells belonging to a frame arrive back-to-back or regularly spaced. Thus, it is the frame delay and its variation that matters, not cell delay.

A frame is defined here as the ATM Adaptation Layer (AAL) protocol data unit (PDU). One problem in measuring the frame latency in ATM networks is that when seen inside the network, the frames may be discontinuous with numerous gaps between the cells as well as cells of other frames. Note that the monitoring equipment, if placed inside the host, will be affected by the performance of the host and may not accurately reflect the performance of the switch. Thus, the test probes of the monitoring equipment should be placed at the entrance and the exit of the system to be measured.

The main goal in designing a good metrics for performance testing is that the metrics should be, as much as possible, representative of real network situations. Also they should be independent of switch architecture and test workload. Based on the above criteria, it is shown in [1, 2] the advantage of using MIMO versus other metrics like FILO, LIFO and LILO. In this paper after introducing a new definition for MIMO we focus more in comparing MIMO versus LCD. Also we show that MIMO can be aggregated and explain the meaning of different contributions in the aggregated MIMO. A clear understanding different contributions to the total latency is very important in designing networks with latency constraints.

2. MIMO definition and zero-latency switch:

One of the metrics used to measure frame latency FILO (first-bit in to the last-bit out) latency as indicated in Figure 1. Other alternative metrics such as FIFO (first-bit in to the first-bit out), LILO (last-bit in to the last-bit out), and LIFO (last-bit in to the first-bit out) latencies can be easily obtained from Figure1. A complete analysis of these metrics is given in [1, 2] where is shown that unfortunately none of the four above metrics is appropriate for ATM networks. So LIFO may result in negative values, FIFO does not reflect the expansion and compression of gabs on output, and FILO is strongly influenced by the frame gap pattern. For this reason we have introduced a new metric to measure frame latency, called MIMO (Message In Message Out) [1, 2, 5].

Figure 1. Frame latencies

MIMO latency is the difference between the measured LILO latency of the frame through the system under test and that through an ideal system [5, 6]:

MIMO = LILO – LILO₀(1)

Here LILO₀is the latency that the frame would experience if it is passed through the best possible (ideal) switch. In other words MIMO latency is the extra delay introduced by the switch under test compared with its corresponding zero-delay switch. For example, consider the frame and the switch of Figure 3. Figure 2 shows the output pattern when the frame is passed through an ideal switch. The ideal switch, in this case, sends out the first bit as well as the other bits as soon as it can. As shown in the Figure 3:

LILO₀= 3c

LILO = 4c

MIMO = LILO – LILO₀= c

For other values of n and m:

LILO₀= n(m-1)c

LILO = n(m-1)c + delay

MIMO = LILO – LILO₀= delay = c

In our case delay = c and MIMO = c

An important point, which needs to be clarified is the influence of Input/Output speeds of the switch. When we change the in/out link speed, we are operating with a different switch. Its measured delay may be very different from that of another switch with almost the same hardware but different I/O speeds. The corresponding zero delay switch is also different. For instance, in the examples given above, LILO with m=2 and m=100 are different. So are LILO₀s. The difference between LILO and LILO₀is MIMO. MIMO definition shows that MIMO metric doesn’t depend upon the measurement loads. This doesn’t mean that "delay" and MIMO value do not depend for internal implementation reasons upon the measurement loads. If there is a dependence of the delay upon the measurement loads this is an issue for the designer and manufactures of the switch.

Our goal by subtracting LILO₀is to subtract out the workload dependent part from the metric so that when comparing multiple switches the results are not overshadowed by the workload dependent part (for example, the part that depends upon n – the frame size).

An equivalent definition of MIMO is:

MIMO = FILO-FILO₀

That is, the MIMO can also be computed as the difference between the first bit in to last bit out latency of a switch and that of an ideal switch.

2. Last Cell Delay (LCD) Metric

LCD measures the latency of a frame through a switch by measuring the last-bit-in-to-first-bit-out (LIFO) latency of the last cell of the frame. For example, consider an n-cell frame passing through a switch with input speed of x and output speed of x/m. Figure 1 shows this case for n=3 and m=2. At input, each cell time is c, while at the output it is mc. In this case, the last bit of the last cell enters the switch at nc and the first bit of that cell exits at (n-1)mc + delay. The last cell’s LIFO latency or LCD of the switch is (n-1)mc + delay - nc or c(nm-m-n) + delay, where "delay" is the real delay introduced by the switch and in our example is equal to c. As shown in Figure 3, for n=3 and m=2, LCD is 2c. For n=100, and m=100, this delay will be 9800c.

So in general for this example we have:

LCD = c(nm – m - n) + delay

As is shown in the above relation, LCD has an explicit dependency from the measurement load: number of frames n and ratio between output speed and input seed m.

If the switch would introduce no delay then this is the case a ideal switch shown in Figure 2.

In this case we have:

delay = 0 and LCD = c(nm – m –n)

Let name the LCD of the ideal switch LCD₀, so we have

LCD₀ = c(nm – m –n)

We can rewrite the expression for LCD as:

LCD = LCD₀ + delay

LCD₀ is a constant factor that depends on the measurement load has nothing to do with the delay introduced by the switch. For this reason it is desirable to remove LCD₀ from the relation used to measure the switch delay, in this case LCD. This is accomplished by MIMO Latency, which definition does contain no terms explicitly depending on the measurement loads. Note that MIMO latency can be obtained by LCD:

MIMO = LCD – LCD₀

3. Accountability

   Generally, the measured performance of a system depends upon the system as well as the workload. Some metrics are highly workload dependent while others are less dependent. A metric, which depends more on the system and less on the workload, is generally preferred particularly if the users are interested in comparing the systems and not the workloads. It turns out that the LCD frame latency as defined in [3] has the undesirable property that it depends highly on the workload. This is obvious from the example shown in Figure 2 and 3. A vendor trying to sell the switch would use small frames, say, n=2, and claim its LCD is zero while a competing vendor will use large frames, say, n=100 and show that the same switch has large delay. Notice that in this example, LCD = c(nm – m –n ) + delay, of which c(nm – m –n ) is the workload dependent part and delay = c is the workload-independent (or switch dependent) part.

   The dependency of LCD metric on the input frame configuration is not a desirable feature in comparing different switches performances. However, if the workload is given and a user is interested in knowing the total delay introduced for that workload, then any of the measured latencies, including FILO, LILO, or LCD can be used. For example, LCD can be used as an indication of delay in an "in-service" measurement where the user is interested in computing total delay between the entry and exit from a given network. This is exactly why ATM forum performance testing spec allows both FILO and MIMO latencies to be specified. LCD can be used in place of FILO.

   On the other hand, for out-of-service performance testing, where a user wants to compare multiple networks or switches, MIMO is a better indicator of the switch performance since the workload part has been taken out.

4. Additivity

   In the references [5, 6], it is shown that MIMO latency of a series of components can be computed as follows:

   MIMO å = å MIMO_i+ å LILO_0i–LILO_{0 å} (2)

   Here, MIMO_iis the MIMO latency of the i^thcomponent, LILO_0iis the LILO latency of the i^thcomponent if it were an ideal switch, and LILO_{0 å} is the LILO latency of the entire series if it is replaced by a black box consisting of an ideal switch. Note that computing LILO latency of an ideal switch requires knowledge of only the i/o speeds and is trivial in most cases. If the input speed is same or slower than the output speed, the LILO latency of the ideal switch is zero.

   LCD is additive:

   LCD å = å LCD_i (3)

   We have not mentioned the fact that LCD for each component is measured differently depending upon whether it is a switch or a wire. For switches, LCD is defined as the LIFO latency of the last cell of the frame. For wires, LCD is defined as the FILO latency of the last cell of the frame. This change in definitions is required to ensure additivity. Note that a wire must always follow a switch and vice versa.

   There are two problems with differing definitions of LCD for switches are wires. First, the LCD latency of a zero-length (or very short length) wire is c (one cell time) and not zero. The latency of a 1 km of fiber would be c+5 m s and not 5 m s. Second, if we put two switches back-to-back the total LCD is not the sum of individual LCDs. Figure 4 shows a concatenation of two switches, with each switch having an LCD of 0, but the LCD of the two switches combined is 2c. In this case, in order to arrive to the right result the user should add the LCD of the intermediate wire, which is defined as the FILO latency of the last cell. Note that this definition is different from the definition of LCD of a switch. The conclusion is that the additivity of LCD is not simple, it requires the use of two different definitions for LCD, depending on the delay element.

5. Simplicity

   Both LCD and MIMO are simple. For MIMO, one has to know the input/output speeds of the SUT and the cell arrival pattern to subtract out the workload dependent part. For LCD, one has to properly classify each blackbox either as a switch or a wire and to ensure that wires and switches alternate.

   In cases, where one is interested in comparing multiple systems, it is important to subtract out the workload-dependent part and so the knowledge of workload is required. However, if one is interested only in in-service allocation of delays among various components for a given workload, LCD may be used.

6. Non-negativity

   MIMO by definition gives always non-negative values. LCD can be negative if the first bit of the cell come out of a component before the last bit goes in. For example, a simple digital amplifier that takes in distorted waveform and outputs noise-free square bit pattern will have a negative LCD.

7. LCD vs MIMO: Further Examples

Figure 5 (example 1 of [7]) shows two switches. The switch in Figure 5a has the simple property that it takes 2c time to process every cell. The input speed is 20 times faster than the output speed. For the 3-cell sequence shown, the MIMO latency is 2c while the LCD is 39c. Of course, LCD does not reflect the true behavior of the switch and can be made arbitrarily large by increasing the number of cells in the sequence.

Figure 5. Comparing LCD vs. MIMO

Figure 5b shows another switch that arbitrarily delays cells by large amounts. The input speed is same as the output. An ideal switch in this case will be able to output each cell as it enters the switch. However, the system under test introduces delays. For a 3-cell frame, the third cell is unnecessarily delayed by 38c. If a customer wants to buy a 20x speed I/O switch this is not a very good switch to buy since the cells are delayed so much. MIMO latency correctly reflects this fact by being at 38c.

When compared the switches in Figures 5a and 5b, it is found that switch in Figure 5b unnecessarily delays the cells and so is not a good switch. Increased value of MIMO reflects this fact. The LCD of the fist switch is 39c and for the second switch is 37c making the customer believe that the second switch is even better than the first one when the contrary is true. In fact, the value of LCD can be made arbitrarily bad (or good) by simply changing the number of cells in the frames.

Figure 6 (example 3 of [7]) shows a concatenation of two switches. The first switch has input speed 100 times faster than the output speed. The second switch has output speed 100 times faster than the input speed. Both switches have the property that they unnecessarily delay the cell by 2c times.

As shown, each frame has 100 cells. MIMO latency for both switches is 2c regardless of the number of cells in the frame. LCD latency for the first switch is 9802c and that for the second switch is only c, leading one to believe that the first switch is really bad. In fact, it can be made to look arbitrarily worse by simply changing the workload – number of cells in the frame.

Another point to note from Figure 6 is that while each switch is good, the system consisting of the two switches together has a bottleneck in the middle. This bottleneck is the consequence of the mismatch among the switches link speeds. If there is no special justification this bottleneck is simply the result of bad engineering. This is reflected by the combined MIMO of 9904c. If the bottleneck is improved the MIMO improves. Combined LCD in this case is 9903c. When there is a link between the switches we should apply the additivity of LCD using the using different definitions for LCD of the switches and that of the wire. In [7] is suggested that the first switch includes the edge links and the second switch includes what remains of end to end connection. This may rise another problem, that of arbitrary assignment of the delay to network elements. For example when the intermediate link is included in the first switch we get: LCD₁= 9902c and LCD₂= c. When the intermediate link is included in the second switch the results are:

LCD₁= 9802c and LCD₂= 101c.

8. Test Measurements with Input Rate Higher Than Output Rate

The test configuration for the LCD and MIMO latency measurements for the case with the input link rate higher than the output link rate, isshown in Figure 7 [1, 8]. It uses a 155 Mbps UTP-5 link between the monitor port 1 and the switch port A1 and a 25 Mbps link between the monitor port 2 and the switch port D1. Figure 8 also indicates the traffic flow direction.

Figure 7. Test configuration for measurements of LCD and MIMO latency

In this configurations:

• Cell Input Time = 2.83 m s

• Cell Output Time = 424[bits] / Output Link Rate = 424[bits] / 25.6 [Mbps] = 16.56 m s

All tests are performed with 32-cell frames. One of the measurements used contiguous frames, i.e. cells of the test frame were transmitted back-to-back. In the rest of the tests, we introduce identical gaps (unassigned cells or cells of other frames) between cells of the test frame.

Table 1 presents measurement results for eight test runs, from which LCD and MIMO latency is calculated. The first test uses a contiguous test frame on input. All other tests use discontinuous frames on input, with gaps between cells of the test frame, as indicated in the second column. Our tests do not show any significant difference if gaps include unassigned cells or cells of other frames, which leave the switch through output links other than the one used by the test frames. The third, fourth and fifth columns present measurement results for the LCD, FILO and MIMO, respectively.

Analyzing the results shown in Table 1 is clear that both LCD and FILO are heavily dependent on the input frame configuration. On the other hand MIMO reflects only the delay introduced by the switch itself. In this case the variation of LCD and FILO could be misleading about the performance of the switch under test.

Note that the switch latency is higher in the first 5 tests due to cell queuing. In the last three tests, the gap between the cells is large and there is no queuing. MIMO latency clearly reflects this effect.

Table 1: (All times are in m s)

9. LCD vs. MIMO applicability

Based on this analysis we can conclude that MIMO and LCD have advantages and disadvantages. These metrics can be seen as complimentary because they are more appropriate in different applications. MIMO metric is better suited for comparing performance of different switches in out-of-service measurements, where it can precisely measure the delay introduced by the switch independent of the input frame. LCD can be used for in-service measurements and for user perceived delay. In this sense, LCD takes the place of FILO or LILO as currently specified in the ATM Forum testing document [9].

Note that if LCD is used in place of LILO, the following relationship may be useful for aggregation:

For any one component:

MIMO = LCD – LCD₀(4)

Note that this relationship applies for both switches and wires. That is, LCD can be FILO or LIFO.

For a network of switches we will have the following expressions:

LCD å = å LCD_i (5)

MIMO å = å LCD_i- LCD_{0 å} (6)

Where LCD_{0 å} is the LCD of the entire series if it is replaced by a black box consisting of an ideal switch.

Furthermore the expression:

MIMO å = å MIMO_i+ å LCD_0i– LCD_{0 å} (7)

can be used to relate the delay of the network to the delays of individual switches measured in performance testing by MIMO_iand to mismatch between input and output link speeds given by LCD_0i.

10. Conclusion:

We have indicated the relationship between MIMO and LCD. One can be calculated from the other. MIMO is better suited for out-of-service measurements where one is interested in comparing multiple switches independent of the workload. LCD can be used (in place of FILO or LILO) for in-service measurement where one is interested in total delay for a given workload.

11. References:

1. Arjan Durresi, Raj Jain, Gojko Babic, "Experience with ATM Switch Performance Testing", ICON'99, Brisbane, Australia, September 28-October 1, 1999 http://www.cse.wustl.edu/~durresi/papers/perfexp.htm/
2. Gojko Babic, Raj Jain, Arjan Durresi, "ATM Performance Testing and QoS Management" in F. Golshani, Ed., "The IEC ATM Handbook" to be published by International Engineering Consortium, Chicago, IL, 2000.
3. K. Glossbrenner, F. Kaudel, "ATM Layer Reference Events for Monitoring Frames," ATM Forum contribution 99-0282, April 1999.
4. K. Glossbrenner, F. Kaudel, "ATM Layer Reference Events for Monitoring Frames," contribution to ANSI Committee T1A1.3, May 3,1999
5. Arjan Durresi, Raj Jain, and Gojko Babic, "Frame Delay Through ATM Switches and networks: MIMO Latency and its aggregation," Contribution to ANSI Committee T1A1.3, May 3,1999, http://www.cse.wustl.edu/~durresi/t1a1/t1a1may99.html/
6. Arjan Durresi, Raj Jain, and Gojko Babic, "Aggregation of MIMO Latency", ATM Forum contribution 99-0243, April 1999, http://www.cse.wustl.edu/~jain/atmf/atm99-0243.htm.
7. K. Glossbrenner, F. Kaudel, "Analysis of MIMO," contribution to ANSI Committee T1A1-3/99-035, July 14,1999
8. Arjan Durresi, Raj Jain, Gojko Babic, and Bruce Northcote "Methodology for Implementing Scalable Test Configurations in ATM Switches, " ICCCN '99, Boston, MA, October 11-13, 1999, http://www.cse.wustl.edu/~durresi/papers/iccc99.html
9. ATM Forum Performance Testing Specification, AF-TEST-TM-0131.000, October, 1999