Welcome To BlogerDesign

Chapter 2, "Packet Switching Architecture," examined how changesin the IOS software implementation have been used to improve packet switchingperformance. Cisco also uses special hardware designs to improve switchingperformance. The first notable example of this was the introduction of the Cbuswith the AGS+ router and its successor, the Cisco 7000 series router, in theearly 1990s. Although these products are no longer sold or supported in currentIOS versions, a large part o

Chapter 2, "Packet Switching Architecture,"examined how changes in the IOS software implementation have been used to improve packet switching performance. Cisco also uses special hardware designs to improve switching performance. The first notable example of this was the introduction of the Cbus with the AGS+ router and its successor, the Cisco 7000 series router, in the early 1990s. Although these products are no longer sold or supported in current IOS versions, a large part of their design was adapted for the Cisco 7500 series routers that are widely deployed today. This chapter covers the AGS+ and 7000 routers because a sizeable part of the 7500-specific switching features actually had their origins in the IOS design for these early platforms.

AGS+ Hardware Architecture

The AGS+ router was an improved version of the original AGS router. The AGS (without the plus) was based on a Motorola 68000 series CPU (M68k) and used media interface cards interconnected by a relatively slow 155-Mbps data bus called the Multibus. Due to the speed limitations of this processor and the Multibus, packet switching performance was limited to the range of 7000 to 14,000 packets per second (pps) even with fast switching—pretty anemic by today's standards. In fact, under some conditions the Cisco 2500 series routers can outperform these switching speeds!

The Cisco AGS+ and 7000 routers were designed before the advent of high speed computer buses and economical RISC-based CPUs. So, when Cisco's engineers started looking for ways to boost switching performance, there weren't many off-the-shelf solutions from which to choose. The engineers decided to solve the problem by designing their own high-speed data bus and packet switching engine. Thus was born the Cbus.

The original Cbus (short for "Cisco bus") was a 32-bit bus operating at 16.67 MHz, giving a total bandwidth of 533 Mbps—a vast improvement over the Multibus. The bus was designed with 32 data lines, 24 address lines, and 8 command lines.

The Cbus switching engine was based on a microcoded 16-bit slice microprocessor with an 80-bit instruction word that allowed several operations to execute concurrently in a single cycle. Unlike general purpose processors such as the M68k, the Cbus processor had a small, customized instruction set designed specifically to support data switching operations. The processor was packaged on a special interface card containing memory for the processor's instructions (called control store), fast packet memory, and interfaces to both the Multibus and the Cbus. This card comprised the switching engine and was called the Cbus Controller.

These two new features, the Cbus and the Cbus Controller, were added to the original AGS architecture to create the AGS+ router. Figure 4-1 shows a diagram of the AGS+ architecture.

Figure 4-1 AGS+ Architecture

The following list describes each of the major components shown in Figure 4-1:

• Processor (CPU)—Runs IOS and performs process switching functions, and performs fast switching between Multibus interfaces. The processor also performs fast switching for packets that are received from Cbus interfaces but can't be autonomously switched.

• Main Memory—DRAM containing the running IOS image, its data, and system packet buffers.

• Multibus—Data bus interconnecting the processor with the Multibus interface cards. The Cbus controller card also has a connection to the Multibus.

• Cbus Switching Processor—Runs optimized non-IOS microcode to perform fast switching of packets between Cbus interfaces. This processor is on the Cbus controller card.

• Fast Packet Memory—High-speed memory on the Cbus controller card that is dual-ported between the Cbus switching processor and the Cbus interface cards. Used to store packets received from or transmitted to Cbus interfaces. This memory is referred to by the name MEMD in the Cbus architecture.

• Cbus—The Cisco proprietary data bus interconnecting the Cbus controller card with the Cbus interface cards.

The AGS+ retained the Motorola 68000 series processor and the Multibus backplane used on the AGS. The Cbus enhancement was made by converting five of the interface card slots on the backplane to access the Cbus. This allowed Multibus interface cards to be used in the same chassis with the new Cbus interface cards.

After the hardware enhancements were added to the AGS+, how did IOS actually use them to increase switching performance? Let's examine how IOS put these enhancements to work.

Packet Switching with the Cbus

Recall the descriptions of fast switching and process switching from Chapter 2, "Packet Switching Architecture." In both of these methods, the receiving interface hardware copies the received packet into I/O memory and interrupts the main processor to begin switching the packet. Whether during the interrupt or in a background process, the main processor has to do all the work of actually switching the packet; the interface hardware just receives the packet and transmits it to the media. Also, packet switching is just one of the many tasks the main processor has to do, so not all of its capacity is available for packet switching.

The Cbus and the Cbus Controller provided another processing alternative on the AGS+. Instead of IOS performing all the switching on the main processor, much of the switching could be performed by the Cbus Controller without involving the main processor at all. The Cbus Controller was capable of performing fast switching autonomously on the switching processor as long as both the receiving and the transmitting interfaces were on Cbus cards. This capability provided a new method of switching appropriately called autonomous switching.

Autonomous Switching

Autonomous switching worked essentially like fast switching—except on a different processor. It was triggered by a packet receive interrupt (this time to the switching processor) and used a local cache to look up forwarding information, just like fast switching. In fact, autonomous switching used the same hash table structure for its fast cache that IOS used prior to release 10.2. IOS maintained both the main fast cache and the local switching processor (Cbus) cache during process switching in the same way. Each time an entry was added or invalidated in the main cache it also was added or invalidated in the Cbus cache.

Autonomous switching did differ from fast switching in a couple ways, though. Autonomous switching didn't support as many protocols as fast switching—only IP, IPX, and bridging—and it handled cache misses differently.

Recall that in fast switching, if a packet is received and there is no fast cache entry for it, the packet is queued for process switching. In autonomous switching, however, if there was no Cbus cache entry for a packet, the packet was sent to the main processor for possible fast switching. If the packet was destined to another Cbus interface, the main processor could fast switch the packet while it was still in the Cbus Controller. However, if the packet was destined to a Multibus interface, the whole packet had to be copied across the slow Multibus to a system buffer, and then process switched. Although the Cbus Controller had a Multibus interface, it couldn't use it to autonomously switch packets.

Cbus Fast Packet Memory

The Cbus Controller design introduced a new strategy for IOS packet buffering that is still in use today on the Cisco 7500 series. On the AGS+, process switching and fast switching used the system buffers for packets as described in Chapter 1, "Fundamental IOS Software Architecture." However, these system buffers held a distinct disadvantage for autonomous switching: They were located in main memory on the main processor card, which made them inaccessible to Cbus interface cards.

To provide packet buffers for the Cbus interfaces, Cisco designed the Cbus Controller with its own local dedicated packet buffer memory. The memory, referred to as MEMD, was a fixed 512 KB region that could be addressed by both the switching processor and the Cbus interface cards. Of the total 512 KB available, the first 8 KB (called page zero) was set aside for control structures, leaving the rest available to be carved into packet buffers.

What's most interesting about MEMD is how it was logically organized. Like the system buffers, MEMD was divided into buffers of varying sizes to accommodate different sized packets. Unlike the system buffers, MEMD buffer sizes were determined by the actual MTU sizes of the Cbus interfaces present on the router—not by a predetermined set of sizes.

The buffers were divided into a maximum of four pools, and then they were allocated to the interfaces from those pools. Within each pool, all buffers were the same size. Due to the limited size of MEMD and its control structures in page zero, the maximum number of packet buffers that could be defined was limited to 470 regardless of the number of Cbus interfaces present.

The switching processor maintained two queues in MEMD's page zero for each Cbus interface: a receive queue and a transmit queue. These queues were similar in function to the input queues and the output queues maintained by IOS. When a Cbus interface detected an incoming packet, it allocated a MEMD buffer from the appropriate pool, copied the packet data into the buffer, and placed the buffer on its receive queue to await switching by the switching processor. If the packet was autonomously switched, the switching processor switched it and placed the modified packet on the transmit queue of the destination Cbus interface. If the packet was fast switched, the main processor switched it and directed the switching processor to place it on the appropriate Cbus transmit queue. For packets that had to be process switched, the switching processor copied the packet to the main processor memory over the Multibus and returned the MEMD buffer to its original pool.

The switching processor also maintained counters and limit values in MEMD for each of the receive queues and the transmit queues. Each receive queue had a receive queue limit (RQL) and each transmit queue had a transmit queue limit (TQL) to prevent a single interface from hoarding all the MEMD buffers.

The RQL value determined the maximum number of packet buffers that could be held on the receive queue at any time. When a receive queue contained the RQL number of buffers, it was considered full and any additional packets received were dropped until the number of buffers dropped below the RQL level. Similarly, the TQL value determined the maximum number of packet buffers that could be held on the transmit queue at any given time. Any additional packets switched to an interface while its transmit queue was full were dropped until its transmit queue depth dropped below the TQL value.

You can find a more detailed description of the MEMD counters and the buffer carving process in Chapter 6, "Cisco 7500 Routers."

The Cisco 7000 Series Router

The Cisco 7000 series router represented the next step in the evolution of the AGS and the Cbus architecture. Other than the mechanical structure and outward appearance, the 7000 wasn't drastically different from its predecessor, the AGS+. The biggest architectural difference was the elimination of the Multibus and Multibus interface cards—only Cbus interface cards were supported on the 7000.

Some of the changes were mostly superficial. On the 7000, the AGS+ processor card was replaced by the Route Processor (RP) card and the Cbus Controller was replaced by the 7000's Switch Processor (SP) card, but the RP and SP retained the same CPUs as their AGS+ counterparts. Although there was no Multibus, per se, on the 7000, the switching processor on the SP still connected to the main processor on the RP via a 155-Mbps Multibus-like interface. These similarities allowed the IOS software and switching processor microcode to work on the 7000 without requiring a large number of changes.

The 7000 did introduce a new hardware feature not present on the AGS+: hot-swappable interface cards. The 7000 routers allowed Cbus interface cards (called interface processors on the 7000) to be removed and re-inserted while IOS was operational with minimal operational disruption to other interface processors. When IOS detected that a card was being removed or inserted, it momentarily stalled operations on the Cbus while the insertion/removal was underway. When the insertion/removal completed, IOS resumed operation of the Cbus and continued switching packets where it left off.

This new feature necessitated a change in the operation of the switching processor microcode. Whereas the composition of interfaces was fixed on the AGS+ while IOS was in operation, on the 7000 that composition could change on the fly. The original switching processor microcode on the AGS+ was designed to carve a static MEMD buffer allocation at initialization time. On the 7000, however, the microcode had to be changed to re-carve the buffer allocation each time a card was inserted or removed. This allowed the SP to keep the most efficient use of packet memory resources even if the interface composition changed.