Exploring the PTX10008 400G Line Rate MACsec 

00:01 hi everyone i'm lester bird from juniper

00:04 networks

00:05 i'm going to show you 400g line rate

00:07 maxsec on juniper's newest modular

00:10 chassis

00:10 the ptx 1008 here's a standard

00:14 disclaimer

00:15 as always this presentation reflects

00:17 current thought but

00:18 we can change direction in the future we

00:20 will generate 400 g

00:22 line rate traffic across maxsec link

00:24 between two ptx 1008

00:27 one of the ptx's runs juno s evolved

00:29 juniper's modern cloud operating system

00:32 the other pdx 1008 runs sonic an open

00:35 source

00:35 nos driven by the open compute project

00:38 we will show three different frame sizes

00:40 66 bytes 1 000 bytes and 9 000 bytes

00:44 for all frame sizes there will be no

00:46 traffic loss showing that the ptx 1008

00:49 can indeed do line rate max x

00:51 for those of you curious why we picked

00:53 66 bytes instead of 64 bytes

00:55 this was an artifact of the tester for

00:57 some reason it imposed a 66 byte minimum

01:01 here's the test topology there's a

01:02 maxsec link between the ptx 1008 running

01:06 juno s

01:06 evolved and the one running sonic we are

01:09 using ixiad to generate bi-directional

01:11 400 g

01:12 line rate traffic before we get to the

01:14 demo we need to set some common

01:16 background

01:17 and nomenclature there are five

01:19 important statistics to look at

01:21 we defined them here first we have

01:23 efficiency

01:24 efficiency is the percentage of the 400g

01:27 bandwidth used to transfer ethernet

01:28 frames but not

01:29 including mac sac octets the layer 1 bit

01:33 rate

01:33 is the amount of the 400g bandwidth

01:36 measured in gigabits per second used to

01:38 transfer the ethernet

01:40 frames again not including the max at

01:42 octet

01:43 when calculating the layer 1 bit rate we

01:46 consider the ethernet preamble

01:48 the ethernet frame the inner packet gap

01:51 but not the maxec header the

01:53 transmission bit rate as seen in juno s

01:56 is called the juno s bit rate it is

01:58 different than the layer 1 bit rate

02:00 it measures the amount of bandwidth used

02:02 to transmit the ethernet frame

02:04 and the maxec header unlike the layer 1

02:07 bit rate

02:08 it does not include the ethernet

02:10 preamble nor

02:12 the inner packet gap the frame rate is

02:14 just the number of ethernet frames

02:15 transmitted per second

02:17 and the frame loss percentage is the

02:19 percentage of ethernet frames lost

02:21 crossing the maxsec link during

02:23 transmission

02:24 if we are indeed sending and receiving a

02:26 400g line rate

02:28 then the frame loss should always be

02:30 zero percent

02:31 let's take a look at the maxsec frame

02:33 format as you can see

02:35 the original ethernet frame is encrypted

02:37 into a secure payload

02:39 maxec also adds a security tag and

02:42 an integrity check value not shown as

02:44 the ethernet preamble and the inner

02:46 packet gap

02:47 the preamble would precede the max up

02:49 frame the internet packet gap would

02:51 follow it so let's start getting into

02:53 the numbers

02:54 the first thing that we need to

02:55 understand is the different components

02:56 that contribute to the bit rates that

02:58 we're going to look at

02:59 there's an ethernet preamble that is

03:01 eight octets and an inner packet gap

03:03 that is 12 octets

03:05 these 20 bytes represent physical layer

03:07 ethernet overhead

03:08 in our case the max overhead is going to

03:10 be 24 octets

03:11 because we are not using the secure

03:13 channel identifier

03:15 if we were to use the sei the maxec

03:17 overhead would increase to 32 bytes

03:20 as we said before we're going to be

03:21 generating ethernet frames of 66

03:24 1000 and 9000 octets the remaining

03:27 fields are just additive

03:28 combinations of the above information

03:30 for instance

03:31 l1 ethernet is the ethernet frame size

03:34 plus the physical layer overhead of 20

03:36 bytes

03:37 similarly the l2 maxx size is just the

03:40 original ethernet frame size plus the 24

03:43 bytes of mac sec overhead and so forth

03:46 for the mathematically inclined or those

03:48 who don't like magic numbers

03:50 these are the formulas that calculate

03:52 those key statistics

03:54 i'm not going to go over each one of

03:56 them but i encourage you to come back

03:58 and look at these formulas after you

03:59 watch the videos they calculate the

04:01 expected values

04:02 note also the ixia field names of these

04:05 key statistics

04:06 we will be examining tx l1 load

04:09 percentage

04:10 txl1 rate tx frame rate the juno s bit

04:14 rate

04:14 is taken from the show interfaces

04:16 statistics command it is not part of

04:18 ixia

04:19 for convenience i've used the previous

04:20 formulas to calculate the various

04:22 statistics for different frame sizes

04:24 notice how the efficiency increases from

04:27 78 to 97 to 99 percent

04:30 as the packet sizes grow not shockingly

04:33 so do the l1 and juno s bit rates the l1

04:36 bit rate increases from 312 gigabits per

04:39 second

04:40 the 398gb per second the juno s

04:43 bit rate which is calculated differently

04:45 grows from 327 gigabits per second to

04:49 399 gigabits per second that's getting

04:51 pretty close

04:52 to our 400g it is important to note that

04:55 neither

04:56 the l1 bit rate or the juno s bit rate

04:58 will actually hit 400 g

05:00 because of overhead of setting the max

05:02 header or the preamble

05:03 or the inner packet gap we can only get

05:06 close to 400 g

05:07 conversely frame rate drops as the

05:10 packets get

05:10 larger this makes sense since a bigger

05:13 packet takes longer to transmit

05:15 due to factors like clock speeds

05:17 latencies when fetching statistics

05:19 and rounding errors observed values may

05:22 not perfectly match expected values

05:24 the observed values will be close to the

05:26 expected values but they may not be

05:28 exact

05:29 in all cases we utilize the entire 400g

05:32 bandwidth

05:33 it's important to realize this however

05:36 because we do not include all headers

05:38 when calculating bit rates neither the

05:40 l1 bit rate

05:42 nor the juno s bit rate will ever reach

05:45 400 g

05:46 finally efficiencies and bit rates

05:48 improve

05:49 as the frame sizes get bigger frame

05:51 rates however

05:52 decrease as the frame sizes get bigger

05:55 as we discussed previously

05:56 we transmit jumbo frames we see that the

05:58 bit rates get close to 400 g which is

06:01 expected

06:02 more of the bandwidth is being used to

06:04 transmit the original ethernet

06:06 frame less of the bandwidth is dedicated

06:08 to protocol overhead

06:09 like sending the maxsec headers the

06:11 ethernet preamble

06:12 and the inner vacuum gap by far the most

06:15 important thing is to see

06:16 zero frame loss when sending a 400g line

06:19 rate

06:20 if we're sending up line rate we do not

06:22 expect any lost packets

06:25 okay we're finally ready for the demo

06:31 we're currently logged into the system

06:33 running junos evolve if we look at

06:35 et000 we can see that's a 400g

06:38 ethernet link we've also set the mtu to

06:41 jumble frames

06:42 if we look at the maxset configuration

06:44 we can see that we're using a 256-bit

06:46 cipher with extended packet numbering we

06:49 need

06:49 xbn for high-speed links like 400g

06:52 we can also see that et000 is set up to

06:55 do

06:56 maxsec encryption

06:59 by looking at the max connections we can

07:02 see

07:03 that encryption is indeed on and that we

07:05 are not including the secure channel

07:07 identifier

07:09 finally we can see that mka the max key

07:11 agreement

07:12 protocol is running on et000

07:20 flipping to the sonic side we see that

07:21 ethernet one maps to et000

07:25 it's a 400g link with jumbo frames

07:27 configured

07:29 juniper is one of the first sonic

07:31 vendors to ever support maxsac

07:33 for mka we created a new max docker

07:36 container built around open source wpa

07:42 supplicant

07:45 when we look at the maxset configuration

07:47 on the sonic system

07:48 we can see that cipher suite number

07:51 three indicates

07:52 256 bit xbn

08:02 we also note that sei is not included

08:05 here as well just as it wasn't on the

08:07 evo side

08:10 if we execute cli we can see

08:13 that these parameters have indeed taken

08:15 effect

08:17 and they do match the juno s evolve

08:19 configuration

08:21 we can also see that mka packets are

08:23 being exchanged with the genos

08:25 side

08:32 we're now on the ixia 400g tester i've

08:35 already set up

08:36 line rate traffic generation with a

08:39 fixed size

08:40 frame of 66 bytes

08:49 as you can see from the flashing icon

08:51 the frame loss is zero percent

08:55 the l1 bit rate is 312 gigabits per

08:58 second

09:01 the efficiency is 78

09:05 and the frame rate is 454 million

09:08 packets per second

09:11 if we go to the evo side and look at the

09:12 maxx statistics we can see that the

09:14 number of encrypted frames jumps from

09:16 222 to 241 billion frames

09:33 we can also look at the genome os bit

09:35 bitrate which we can see

09:37 is 327 gigabits per second

09:42 the data for 66 byte frames was as

09:44 expected this is actually the hardest

09:46 test

09:46 because it has the greatest protocol

09:48 overhead let's try

09:50 a larger frame size 1000 bytes note that

09:53 we are still running at 400g

09:54 line rate and the frame size is 1000

09:57 bytes

10:00 we start the traffic

10:09 as expected the frame loss is still zero

10:12 percent

10:14 the l1 bit rate has now jumped to 390

10:16 gigabits per second

10:19 the efficiency has jumped to 97

10:23 and the frame rate has dropped to 47

10:26 million packets per second

10:29 if we look at the juno s bit rate on the

10:31 evo side

10:32 we see that it has jumped from 327 to

10:36 392 gigabits per second

10:42 finally we change the ethernet frame

10:44 size to 9000 bytes to test jumble frames

10:50 as you can see we are still running at

10:51 400g line rate

10:54 and we're generating 9000 by ethernet

10:56 frames

10:58 starting the traffic we check on the

11:02 frame loss

11:05 and happily it's still at zero percent

11:07 loss

11:11 the l1 bit rate is now nearly 399

11:15 gigabits per second

11:18 the efficiency is now 99.7 percent

11:22 and the frame rate has dropped to 5

11:24 million frames per second

11:27 if we check the judo s bit rate on the

11:29 evo we see that it is now

11:31 399 gigabits per second as well from

11:34 this screenshot we can see how the junos

11:36 bit rate

11:37 increases as the packet size increases

11:40 this is also true

11:41 of the layer 1 bit rate if we track the

11:44 ixia data

11:45 hopefully you notice the most important

11:46 piece of information at all three frame

11:48 sizes ranging from 66 bytes to 9000

11:51 bytes

11:52 the ptx 1008 did not drop a single frame

11:55 when doing max sec there you go the bt

11:58 asic is the high performance networking

12:00 engine that drives the ptx 1008 modular

12:03 chassis

12:04 maxsec is integrated natively into the

12:06 a6 unlike many competitors

12:09 juniper maxsec does not utilize external

12:11 phy devices

12:12 the bta6 can perform inline maxsec for

12:15 all packet sizes at 400g

12:18 line rate if maxec is not needed bt can

12:20 turn off the maxep block to reduce power

12:22 consumption

12:23 finally juniper maxx supports several

12:25 ciphers including

12:27 gcm aes xpn 128 and gcm

12:30 aes xbn 256 as you saw in this demo we

12:34 did the xbn 256

12:36 cipher say hello to juniper's ptx 1008 a

12:39 system that can meet

12:40 all your 400g high-speed networking

12:43 needs

12:44 the hardest system is the bta6 the pdx

12:46 1000a can do 400g

12:48 line rate max sec as you've seen you get

12:51 high speed link security without

12:53 traffic loss and finally you can pick

12:55 your nos you know s volunteer sonic

12:57 the ptx 1008 supports both operating

13:00 systems for maximum flexibility

13:02 here are some references for the ptx

13:04 1000a and

13:06 it's big brother the ptx 10016 i refer

13:09 you to the product data sheet

13:10 we also have some ocp videos that we did

13:13 showing sonic running on the ptx 1008 i

13:16 think you'll like those too

13:18 please check out the ptx 1008 and enjoy

13:21 the full power

13:22 of 400g stay healthy and stay safe take

13:25 care

13:26 bye