Juniper and the Marvis Journey 

0:08 so bob friday and now the chief ai officer of juniper in addition to being the cto of aid president and sedira says

0:14 mobility phil day has always been this special event for us so it's great to be back you're part of

0:19 the family right wireless is a very small community networking's a very small community and so it's been great

0:24 to be working with you you know when susan and i left cisco it

0:30 was kind of with the thesis that cloud was fundamentally a better way to maintain

0:35 and develop software and i think we've all found that that is true i think the other thesis when we started

0:41 this was around aiops that fundamentally we're moving from a paradigm

0:46 of helping manage network elements to one to helping manage this client to cloud experience

0:52 and that was the other thesis amist and what you see here is this is the same slide this is my fifth mobility

0:58 fill date i think besides the colors the slide has not changed i think you guys have seen this slide

1:04 how many people have seen this slide five times in a row now yes tom but what has changed is we have been on

1:11 the journey to build an ai solution that can manage and operate networks on

1:17 par with i.t human domain experts now what has changed on this slide is

1:23 really where are we at that journey you know since we've joined juniper we've basically extended marvis

1:30 across both wireless wired and route now and when you look at the journey this

1:36 journey basically starts with data you know what people always say you know

1:41 ai requires great data you know those who know me i make this one barrel of wine here great wine starts a great great

1:48 same with ai data and what we're doing with the data here is we're really starting to extend

1:54 the visibility of marvis and so what you're going to hear today is really how we're extending the visibility marvis

2:00 out to windows laptops we're starting to bring this client's view the network back into

2:06 marvis right and so that lets us start to answer questions more granularity now let's stick it down to what driver if

2:12 there's driver problems in this device we now have visibility what's going on from the client view the network

2:18 the next thing we're starting to do we're starting to ingest data from outside

2:24 the juniper domain right this is sentiment analysis this is starting to ingest data from if windows

2:31 is having a problem or teams are having a problem what we want to know is is that because some service went down did

2:38 aws go down the teams go down so now we're starting to ingest data from outside the network

2:44 to give us more visibility when there's a problem you know is it somewhere outside of our domain so that's the big

2:50 thing you're going to hear today is really what we're doing around the data side on the primitive side this is

2:55 fundamentally and what we learned amazingly what we learned at mist was even before we got to the ai ops

3:02 customers just appreciated the fact that we had data in the cloud right it was the first time that

3:07 customers actually had visibility you know the same data we need to get to marvis is actually the same data that any it

3:14 person needs to solve a problem so this phase part of the architecture

3:20 is really getting data back to the cloud and into some format that we can apply some ml to it

3:27 and what you're going to see here new to us is basically we're starting to extend this to the missed edge right this is

3:32 the this is the uh the aws outpost right when we have to put

3:38 compute storage on site you know this is like that tunnel termination this is the one last piece

3:44 of the controller that people like right we moved almost all that controller to the cloud

3:49 except for the tunnel terminations people still like to terminate the data plane on site and so that's what allows the missed

3:55 edge due to that and then the second big thing is really we're starting to bring the land

4:00 into marvis and so you're going to start seeing more land data come back into the cloud for

4:06 us now probably the third big thing is around this data science toolbox you know people argue we all are using

4:13 the same algorithms right there is no real magic here the real magic here is the team and

4:18 you'll be hearing from g shane here who leads our data science team and as suje says

4:24 we can all use the same algorithm that you cannot replace the people you know what miss is bringing to the

4:30 table right now is over a 14 plus eight years but into eight

4:35 years into the adventure so now we have eight years of experience and we have probably the best data

4:40 science team right now working on networking problems and this is what you're seeing here

4:46 right now and probably the fourth piece is really around conversational interface and this is what i tell you is around trust

4:53 you know if you're going to build an ai solution that's going to do something on par with an ip

4:58 that ai solution has to gain the trust of the ipt the it team needs to be able to interact

5:04 with that ai solution like you would a normal person right you want to be slacking you know you

5:10 want to feel like you're the human on the other side of that slack and that's really what you're going to see around the conversational interface

5:16 and where you're going to see today is really how we've extended that to full stack you know seven year girls we started with the access point

5:22 now you're going to see that go all the way from the client all the way to the cloud you're going to

5:28 see that conversational interface start to interact but you know if there's a problem we can actually graph that all

5:33 the way to the end and finally this is where the magic is right

5:38 you know this is the being okay you know there's a problem what are you going to do about it

5:44 right it's one thing to have observability that's another thing to actually take action actually do something about it

5:50 and this is where we are now starting to get to the point where if we find a bad cable

5:56 you know does the i.t department trust marvis now to issue that support ticket

6:02 you know there's plenty of things where we do take care of rm and we proactively take care of things in the background that people don't even know about but

6:08 there's other things where we actually have to have support tickets right you know have to replace a cable an ap

6:14 this is really where the magic is starting to happen is when you start to trust your ai ops

6:19 ai assistance solution to actually issue that support ticket like a human would do

6:26 and that's where we are on the journey right now and we're just starting to begin to see that happen right right

6:31 right there actually people actually trust marvis to actually start issuing those support tickets as they would an ip person

6:39 now the only thing i would say this is the toolbox you know i'm a big believer in truth in

6:45 advertising you know so behind every one of these ml algorithm is something a user experiences right

6:52 but the point of this slide is really we're trying to build something to something on par with a human right

6:58 and this is marvelous at the end of the day this is what we're trying to make happen

7:04 now under the hood this is all the stuff that g shannon can take care of right all this stuff it takes all this stuff

7:10 to actually do something on par with a human and that's the point of this light it's not one magical algorithms it's like

7:16 your self-driving car there's a whole bunch of things going under that tour to get that car to drive itself

7:22 this is probably the thing i am most proud of and i was this thing i would claim we missed

7:27 are the only ones in the industry this is the one differentiated that cisco aruba no one else is doing i

7:35 would challenge you to go ask them we actually use at mist

7:40 our customer success team right this thing is tied to the hip those guys go over support tickets every

7:47 day because our second when you get to the cloud that success team is a proxy for our biggest customers

7:54 right that success team has visibility to every big customer network we are

7:59 running right now if you can make this a sex team happy ultimately we're making our customers

8:06 happy you know and when i hired ginseng it was the the deal i made with him is like hey

8:11 if he can get marvelous to answer 90 percent of support tickets 80 90 was some number

8:17 a big number you know that i'd go help them start another company

8:22 or something like you know we're getting closer so we're up like 75-80 we're almost to the point where marvis can now

8:29 answer 77 of all these support tickets coming in coming into uh the support team and the

8:35 point of this is in addition to having to build this real-time pipeline and all the other

8:41 stuff this is an organizational issue this is the other thing you know where you find out big companies can't do this

8:47 this is another reason suzuna left cisco is because organizationally you have to

8:53 tie your data science team your success team that makes sense and what that results in

8:59 is this this is the journey we've been on the ginseng team have been on right

9:05 this represents how often marvis was able to help the successfully how often was marvis

9:12 able to get the right answer you know on these support tickets and what you find is it took us probably

9:19 a year just to figure out what data we needed right we had to get data back to the cloud

9:24 you know so once we got data back to the cloud then the data science team could start working on it and actually making progress

9:31 and so this basically shows you where we are on the journey and i would tell you this is the one differentiation

9:38 that we still hold while we all have the same data science toolbox no one else is eating their own dog food

9:45 like we are and until you do that you will not get on this path until you actually tie your

9:50 data science team to your customer success team if you're going to be spending your wheels

9:56 one of the things that i i don't think people appreciate is is actually you know the curve becomes

10:04 harder every week when we solve a problem teach marvis how to solve that

10:09 then customers can self-serve within marvis and solve that problem or we've self-driven the problem out and and and

10:17 g is going to talk about what self-driving networks mean and when and how do we do how do we do this so what

10:22 is the outcome of this why should customers care about this why are we so passionate about this one chart and

10:29 graph let me tell you this one thing in the last year or so you know probably

10:35 you know through covid and everything our number of devices on our cloud has more than tripled right

10:44 that means the number of customers you know working off of the miss cloud continues to sort of exponentially grow

10:51 our incoming support tickets on our team our incoming support tickets while our

10:57 business has tripled has barely budged meaning less than 10 percent increase in our support tickets

11:04 coming in from our customers so i always say you know if you think the product of missed is awesome the people of mr

11:10 amazing you still have not met the best part of mist which is our customer

11:16 success team because it is backed by ai and g xing and team truly

11:21 this is why bob is so passionate about the team structure the organizational structure the large companies cannot you

11:28 know really build themselves around physically within the office of mist our

11:34 customer success team sits at the epicenter of our office because that is how important we believe ai can serve

11:41 our customers so with that g-shang uh well let's take them to uh self-driving network fee yeah thanks leo and

11:48 this is really the high-level architecture of marvis that explains the secret sauce you know what bob just

11:55 described why marvis is getting better how what do we build to really make marvel's getting better you know day in

12:01 and day out you know if you see on one side marvis now is taking almost the

12:07 petabytes of data every day across the wi-fi wide and van and the clients and

12:13 through this really this you can see the hierarchical ml pipeline we build in the middle

12:18 so the key there i want to highlight is the resident generating terms of the events and the alerts just telling you

12:24 this device is down or not marv is really focused on the root cause of the probe a simple example you know uh

12:32 rather than telling you ap is the drop in package you know have ethernet arrow connects a client's fair connection with

12:39 iap mavis really puts them together use some advanced ai ml technology to really kill

12:47 you the root cause it's a bad cable you know rather than give you like tens of the alerts and events we all put them

12:54 together consolidated into one action called the battery cable so on the right side you see what the

13:01 markets deliver even given all of the complex work markets i'm doing across such big amount of data but what we want

13:08 to deliver to the customer is really is impressive it's just some simple actions you know

13:14 the ultimate goal of marriage is what we call zero support ticket in other words we want marvels to detect

13:22 and remediate the network problems before any of the user compliance to their ipt

13:28 so that's all about the self-driving actions we deliver only for the remaining like network problems which

13:35 are not in our control or marvels does not have the high confidence takes action

13:40 we have to give you the detailed guidance and also the insights to help you to

13:46 really highly reduce the response time to you know in terms of getting the root cause themselves

13:52 so i think uh still you know i will keep to here and our uh is going to play a quick demo really

14:00 about what is called the marvel thing action so what's the demo about right

14:07 essentially so far you heard bob talk about marvis and trust you heard sudhir

14:13 and bob talk about the efficacy of marvelous and answering questions what we are going to unveil today is exactly

14:20 what marvis is thinking what marvis is looking at what marvel's literally watching when it's recommending an

14:25 action or it's remediating your network automatically so this is one of our

14:30 large customers obviously we're not going to share names but trust me it's a very well known customer um and what you're seeing here is a blip

14:38 on the capacity smd in the wi-fi world as you know we've come a long way from coverage designs it's all about capacity

14:45 it's all about users connecting the network making sure they're getting the capacity that they need what you're

14:50 seeing here is when the day began the capacity sle for this customer at this large site with 2000 users connecting to

14:56 the network was at a not a very good success rate right it kept dipping

15:01 dipping dipping and it actually reached that 79 number that you see so what's happening in the back end so

15:07 marvis is watching this trend and marvis is then trying to catch the anomaly

15:13 that's driving the capacity to go down is it because of the load is it because of interference

15:19 wi-fi non-wi-fi what is it so here for the first time are we giving you

15:24 what we call the ai logs of what marvis is looking at what maurice is thinking and more importantly what marvelous is

15:30 doing in the back end to remediate your network so gishan if you want to talk about some of this yeah i think this is

15:36 if you see here is really this is the full cycle of the ai we talked about it's the starting from the monitoring

15:43 then the detection the beep right then it's really automated troubleshooting we try to find the root cause of the

15:49 problem and describe and now it comes to the action piece maurice is confident it's because of the

15:55 interference so marvel's trying to change the channel and finally it's a very important piece a lot of people

16:02 forget or even miss in the ai is a validation how do you know your action is really

16:10 ineffective so this monitor detect action and validation this builds the

16:17 full life cycle which is what we call you know in the ai world is the reinforcement learning

16:22 you have to continuous reinforce the ai is doing the right thing and it

16:27 continues getting better so what began what began as a 74 percent fail turned

16:33 to a 97 success even though when we had peak usage before the client count went down and what we are trying to show you

16:40 is it's not just magic that's happening behind the scenes there's an actual anomaly analysis that's saying this is

16:45 not uh you know a frequent daily trend that happened this is truly an

16:51 anomaly where the capacity is dipping and what is the process marvis goes through to remediate that automatically

16:56 for you you had a question my question was um how does marvis knows when to take

17:02 action like how long does it watch the trends going down for and then the second question is

17:08 related to that you know um for the validation process how long does it perform the validation process for

17:15 uh very good questions first how does maris know this is really the detection

17:20 piece we have built actually some neural network based anomaly detection which

17:26 continues morning the first the baseline of the site you know it's not that easy for a lot of sites they'll see the

17:33 narrative right there's no client during the night there's a busy client so this graph is not black so first we have to

17:39 use the 30 day of the history data to build the memory of that site for the in

17:45 addition we also use the pure baseline because especially for new side or side to which you

17:51 have like a bigger fluctuation so you have to borrow the like we actually use

17:57 the all of the size from the same off even the different companies which are in the same back uh

18:04 industry vector to build this memory of each of the sites

18:09 then we will compare in the live we will compare you know the real time what is the current value compared with what is

18:16 the a i predict the value at this time of the day that's the whole like uh so would we say though it's less than an

18:23 hour though in terms of protecting an anomaly and then taking action on it so right now we're aggregating data on 10

18:28 minute buckets yeah yeah this is like tens of you know probably tens of minutes by the time you know

18:33 there's an anomaly you know until you detect it that something's gone something's happening

18:39 yeah i might have missed it but no sorry the second question the validation also happens at the same frequency you know

18:45 this is almost like every time we decide whether there's anything wrong then the root cause analysis is almost instant

18:52 then after that if ai decides to take action you will see actually the really

18:58 the effect after the next 10 minutes so the key part is we're always baselining we're always learning it's not like when

19:04 the anomaly started now that's when i start learning and then i go back in time and look at data every minute every

19:10 day of the week every week of the month we are learning so if the anomaly trigger actually happens within minutes

19:16 when we say oh look at my last week look at yesterday look at the day before this trend wasn't happening my capacity was

19:22 pretty solid but something happened and that triggers the marvelous audit log saying okay this needs to be remediated

19:28 yes so how does that uh baseline work for let's say a campus that does uh graduation once a year where it's

19:35 outside of that 30-day window there's going to be a huge spike in say guest users that are coming on campus

19:40 great question so yeah that's that group that's where the peer

19:46 comparison comes from the picture if you see this is the behavior of common behavior across all of the campus right

19:52 campus side so when we look at one campus it looks like a bigger normally but when you really look at the compare

19:58 with its peers then you know this is even we don't know what's happening but we know this as normal because there's

20:05 no chance all of the network from this uh all of the separate campers all have

20:10 showing the same tip at the same time this is where the peers the more data why we the more customer we have the

20:17 more data we accumulate the more intelligent does this the ai engine yeah the one the one thing i might add to

20:22 this thing there is the one the one thing i have found with the one thing that's changed

20:28 you know we know what the convolution networks did for image detection right that just changed the whole world on how

20:34 we well we could do in our networking world these lstm networks have changed the anomaly detection game

20:40 to be much more accurately so to that problem these lstm networks are predicting both also what the next 10

20:46 minutes should be it's also predicting the confidence interval and that's what's getting the low false

20:52 positive now because we're predicting two things now we're not just predicting what we think the next 10 minutes should be as those con as those graduates come

20:58 into the network we're also predicting what we think the the anomaly confidence interval is and that

21:05 is why baseline changes and that is what's changing the game on this anomaly detection after 20 years

21:11 right it's really these lstm networks are really making a difference for networking marvis saw the issue

21:17 triggered it made a decision and then you followed up with validation right

21:23 what if what marvis did caused something else because

21:29 you you might have fixed the problem that you were trying to hit and cause something else at the same time so how does that how does it make that

21:34 validation and still continue to watch other things that's a very also very good question

21:40 this is the what we call the really the punish and the reward in the

21:45 reinforcement landing you're totally right ai is not always correct that's the reason we very quickly iterate

21:52 you know it's not the key is not always what make 100 correct decision the key

21:57 is literally and validate and change so that's the reason why sae this time

22:04 series like a real-time live like monitoring of the network continues feedback to the model that model knows

22:12 whether it's making the change on the right direction or not and change that correctly and another point to add to that keith

22:19 is yes we showed you the example of the capacity sle so the users who are experiencing

22:24 a non-optimal capacity metric but while marvis is making that change reacting to

22:29 capacity it's also looking at the other three connection force connection vectors so it's not like because we fixed capacity here we sort of impacted

22:36 coverage or roaming we look at that or rather maris looks at that in conjunction to the capacity to make sure

22:42 that we're optimizing all the sles and not just the one we're focusing on but we're remediating for the one that needs

22:48 help i would i can't help but jump in yes because the other big change that happened you know 20 years ago at

22:54 airspace we were trying to do this on uh ap to ap right we're trying to minimize

22:59 interference and power and all this stuff what has changed here is now we're doing these reinforcement that's looking

23:05 at the user experience right and so that's probably the other big change that's happened you know when i was doing aerospace it

23:11 was all about ap data this is all every data we start is the user experience we're basically looking at the actual

23:18 user connectivity experience so in this case we're looking at asymmetry as we start to screw around with things or

23:23 change knobs you're actually monitoring that user experience real time across the board right and that's the

23:28 reinforcement learning so i guess if we turn the knob this way hey we also see the users are getting worse

23:34 you turn the knob back right thanks i have a question for you

23:40 at the beginning you said you were starting to integrate third-party algorithms into this right

23:45 so how do you yeah so if in terms of data but how do you actually incorporate that

23:51 and without gaining the expertise of that product or that knowledge that that

23:56 third party vendor has that's a great segue to the next slide thank you very

24:02 know much should know we spoke before though yeah no we didn't seriously we didn't this one

24:09 this is another really very exciting and i will see first time in the industry

24:14 about applying the state of art graph technology into network industry solving

24:20 complex world this diagram may remind you this like an announcement two years ago

24:25 also in mfd that we are a lot of the first vendor in the network industry

24:31 that applying the state of our graph technology into solving you know those

24:36 this client cloud complex problem you know this include like you see here a very large scale distributed graph with

24:43 the billions of the node and the age we accumulate so far and some of the cutting-edge graphic ml models like

24:50 graph neural network or not so you may wonder i think this is very

24:55 exciting and this becomes the foundation of our client-to-cloud troubleshooting so you may wonder why we are so

25:02 believing into this graph technology we have been continuous like pushing the limit of this mission the last two years

25:09 with certain additions every year so it's just like a gardener set you know

25:14 aiols when you're talking everyone is talking about aio see how much data we collect how many of the like customers

25:22 we have it's not about how much amount of data you collect these days

25:28 it's about how you really connecting these data points how you organize the data how you digest the data finally get

25:35 to the root cause we talked about and we believe you know as you know network is itself by nature is a ground

25:43 right it's a very high dimensional fully connected world that's the reason we are very keen and

25:49 we are really pioneering this graph technology into the networking and certainly we also suggest you know this

25:56 is a very powerful weapon that every vendor needs to you know leverage

26:01 so come back to your question and really we're very glad here after two years you know every year there really is a big

26:09 step forward in this graph remote is to take in the third part so the biggest

26:14 value add if you look at the right side of this client cloud you know because most of the juniper and mr deployment

26:21 cover the lamp and the vamp part of your network but a lot of the time with

26:26 the really the sas application the user experience problem can be very often caused by the service provider

26:34 all the really the sas application itself right so the third-party data here really helped

26:41 us cover the last mile gap in mars in terms of this client to cloud

26:47 like vision and you can see here so the way we really consolidated is you know we built

26:53 that if you remember that the pipeline that hierarchical data funnel we build

26:58 with the streaming batch data processing and we are just it's the same pipeline we built for

27:05 the bank and now third party you can think that's really all of the magic happens we consolidate all of the you

27:12 know all of this connectivity information with those events information

27:17 you know all of the data points through that funnel and finally use some of the mai technology to get to that

27:25 make sure we answer does that kind of answer your question about the data sort yeah so i get the data source but the

27:31 knowledge of what to do with it really exists in the vendor and it's very hard to

27:36 then i mean if some of that knowledge is proprietary or you know and intellectual

27:41 property so how do you bring that into an ar so for us our grounding our foundation is the user experience so

27:47 what we care about from these third-party source data sources is how is that data point impacting my user so

27:54 when i'm doing a mean time to innocence of okay it's not the network it's something else i should be able to

27:59 pinpoint what is that something else and that's where the third pie data ingestion comes in so we're not solving

28:06 third party issues but we are trying to give you a full pipeline of root cause

28:11 analysis proactively to say hey in the pipe of a client application going all the way to a cloud or a data

28:18 center where in the life of the packet did the connection fail yeah so where in

28:24 the life did the experience be yeah so the data we're pulling in is really answers almost root causes like these

28:29 this is data like zoom will give us data that tells us why zoom is having a problem or this this is data we're

28:35 getting from like a service fighter or a an aws service right where you want you

28:40 where you where you want to know is is aws up and running are we dealing with a cloud outage which is not you

28:47 know so when someone has a problem that's the type of data we're getting back from these third-party sources now

28:53 okay yes that makes sense just a follow-on question if i may so um one of the things that's really difficult to

28:59 troubleshoot is when you have a multi-vendor environment you know and you have a problem you know who do you point the finger in there's a lot of

29:05 this going on is marvis gonna provide data that can help me prove that this

29:12 vendor's problem and and can you talk a little bit about that yeah i mean so i would probably the best

29:17 example is you know dhcp where you may have info blog you may have a dhcp vendor right rip vendor

29:23 right and so we know that that the problem is there at that vendor at that point of the problem whether it's a dns

29:30 problem right so the goal is to basically extend marvis marvis's ai ops right it really

29:36 doesn't care what's underneath it whether it's juniper cisco or vista you're basically trying to get to the root cause of some element and what you

29:44 find is those elements may be you know they'll be all over the place right maybe the cloud provider dns dcp

29:50 there's all types of providers you keep a network and further to that point apart from dhcp dns radius isp

29:57 application right from a multi-network network perspective as much as we would like the entire stack to be juniper we

30:03 know we're going to get into account before we convert them to full stack to have this wireless and something wired and

30:09 some something went right even with that with with the ap is plugging into third-party switching we're still able

30:16 to give you root cause identification in terms of bad cables or missing vlans so

30:21 there's still indication obviously we can't fix that for you because we don't control the multivita network but we can

30:26 still identify the root cause for you in fact even on the land side if all i have is wireless i can still tell you if

30:32 there are authentication issues is that because of something on the dhcp or on the radius server or is it because of

30:38 something changing on the van where the cert is not going to and we had this actually happen when the one of our large details in the deployed where it's

30:44 a multimeter network it's a cisco wired it's a

30:51 cisco wan and they have a missed wireless

30:56 they were having issues in the wireless it clearly wasn't a wireless station so we want to give them the evidence of

31:02 that it was actually something on the van causing devices certain devices not being able to connect but we were able

31:08 to give them that indication hey the issues on the van side because the search are not right but that's why so that's what again it all ties back to

31:14 the end user experience and that's why it gets back to the kind of customer success data science tied together

31:20 this third party effort is being driven by customers right you know for producing to get up to like 95 percent

31:27 you know we've taken care of all the easy stuff right wireless switch when right now it's down

31:33 to okay what else i want to get to a hundred percent now i got to go outside yeah to get to the next level root

31:39 causes oh great yeah i have a question from someone online they asked how you would integrate with um

31:46 organizations that have strict change control so if marvis is recommending a

31:51 change and then implementing the change do you are you able to tie that into some sort of ticket system and then have

31:58 a manual approval and then have melorvis integrate that change great question so marvish today all the actions actually

32:05 do tie into a ticketing platform so it depends on again goes back to trust as bob was saying do you trust marvis to

32:12 make the change for you or do you just want marvis to identify the issue and then open the ticket for you and let the

32:18 admin take the action right so you have if it's a juniper network where marvis can make the conflict change for example

32:24 again a missing vlan a negotiation mismatch you mars will ask you hey here's the issue i found it'll create

32:30 what we call an ai ticket for you it can then create a web book alert into your ticketing platform like a service now or

32:36 whatever you may have via apis and then it's totally up to you where you say hey marvis go fix it and mrs talks to

32:42 configuration enables that workflow or the admin goes to a change control window and does the action so a lot of

32:48 these things that impact data plane like a missing vlan or like a negotiation mismatch or

32:54 other things are admin led admin authorized but like an rm or a capacity sle that

33:02 marvels a lot of media because there's no action to be taken but it's a network config that marvis can look at and take

33:07 action on automatically yeah the other point sorry i just want to add here is really we want to put a

33:13 mag in the middle for the ai because this is really the feedback we will talk about right

33:19 whether the user really takes the action as marvel suggested or use the terror

33:24 marvels no it's because some of the special setting in this store that this is the first party for marvin this

33:31 actually is the most valuable like feedback we have to collect through this man in the middle ai process yeah i had

33:38 a question for the third party uh like you getting data from isp and and cloud providers is that something that

33:45 just that you guys do in the back end so you you connect with like you know zoom and and teams and isps or is it

33:52 something that we have to configure as the network operator to tell you guys we're using that isp that service like

33:58 how do you guys handle all of that data in jest so in short the answer is we learn right we

34:05 learn your applications we learn the destination and again if you have the sd one piece as well we know the circuits

34:10 we know the provider so we can get that intelligence ourselves what we do ask you though in based on certain use cases

34:16 which applications do you care about do you care about the guest user traffic that's doing youtube videos and hogging

34:22 your network or do you care about your sas apps your office 365 your sales etc

34:27 that you want us to put more focus on and optimize for and then that's where we'll build the bridge but the

34:33 integrations happen in the back end based on what we learned about it yeah you don't have to config give you one

34:39 example service provider right from the we know the from the ip we know what the

34:44 s7 and which service provider you use like any country in the world you know that no no no i started with the us for

34:50 the real estate so let's go into the demo but great question yeah so this one essentially we're look we're

34:57 integrating with the sentiment analysis tool looking at all the isps and we are asking that same question hey what's

35:03 happening with my team call right so here mark's looking at exactly which team calls which session you're

35:09 interested in and it's now showing you the end-to-end view of not only just wireless wired van you already heard

35:14 about that it's looking at the actual isp linkage as well right and i'll tell you at each step of the way what exactly

35:21 is a problem is it a wireless problem is it a van problem or it's actually isp on reachability so

35:28 here we figured out it's an atp circuit and we're giving you the health of the atnt circuit as part of our third party

35:33 integration that's how intelligent marios is becoming it's nice

35:41 because this is really the really the power here is a single pound of glass you have really the client q cloud

35:48 of your whole network troubleshooting within this small like a conversational system

35:53 so we could think of if we think about the api we could have a system where marvis opens a ticket with

35:59 a service provider automatically on our behalf potentially someday

36:05 but we don't want the sps to hate us either but on that note of data integration royal sorry last question

36:12 on the third party integration do you have a set of requirements for third-party integration or is there some

36:18 sort of initiative to standardize what type of data you would ingest like

36:24 shared data that way it's more standardized across all third-party devices we are at the mercy of what the

36:31 tools provide so we look for the data set that we need like the first one is this availability and then we use that

36:36 and that every vendor has that right so as we expand potentially yet but right now just hey is the service available

36:41 pull that in and then use that in markets but talking about third parties if i can if i could just add to that i

36:48 think the whole notion ravel and francois here is is that we do not want users to have to know

36:56 what they need to integrate that's the whole point here that's why we are saying third-party integration where

37:02 we're going to learn and continue to make it better with zero user

37:08 configuration required that's the initiative here truly um we want to make

37:13 sure if you go back to that client to cloud the last picture today no vendor

37:19 literally zero vendors not cisco ruben rocky here you know a wrist and nobody

37:24 can show you and connect the dots across wi-fi switching

37:29 sd-van and application including now with this third-party

37:35 integration the isp services this is where all of the ai mumbo java we're

37:41 talking about is coming to life and we don't want the user to have to know and configure right and friends what to your

37:47 point yes we're starting with some us service provider integrations but you know going forward or learning not

37:54 integration um but you know this the idea of this is that the user doesn't have to lift a finger so if you look at

38:00 it here i think we are running over time just the last comment it's really you know if you think of the backhand these

38:06 are billions of the node and age graph we keep but what we want to final deliver to the

38:12 user is just that simple like interface for trouble so this is the true power of what the ai

38:19 should bring to the real world and you'll hear us talk a lot more about the conversational assistant how it's

38:25 truly helping operations team across the world across all verticals and becoming

38:30 the best friend but it's all about making that network operation simpler but talking about third-party data

38:36 ingestion we've also got a new announcement around expanding our location

38:41 enabled marvis client now also serving the needs for windows laptops

38:48 so we have now made available a windows client sdk that can get telemetry

38:54 information from windows devices windows 10 windows 11 and answer the

38:59 the two key questions that always come up hey is the driver the problem when a device is not able to connect the

39:05 network of having a bad experience yes how quickly does it pull or like how quickly is there it's constantly sending

39:11 data to the cloud so it's almost every second so the the two key things that we try to

39:16 solve again data is data but the question is what action can i drive from it so the two key action we're trying to

39:21 drive from it is how is the client driver itself contributing to the client's ability to

39:28 connect on the network or roam the network and therefore how is the roaming right today we do an

39:35 awesome uh job of looking at the roaming and classifying a sticky client or a

39:40 slow roam based on what we see as telemetry data coming from the aps we are now able to get higher efficacy and

39:47 even more granular on that front of a successful rom or even coverage holes by getting that data directly from the

39:53 client so not only do we do androids and the devices we are now announcing the windows client

39:59 as well so just a brief demo of this so now in

40:05 the mist portal and if you all would like to try it out let us know we make it available for you it's a windows apk that can get pushed

40:11 by your mdms onto your onto your laptops like any other application once it's there it starts working

40:17 automatically it makes a connection to the juniper miss cloud sends information

40:22 so now when you go to client insights you will be able to see the rsi not just from a network perspective but

40:29 also from a client view perspective and that is where the symmetry comes into play yes is that agnostic to

40:37 the client connecting to any network or is that specific to your network like if i go and visit another organization that

40:43 happens to have missed and i have this agent loaded on my laptop are they going to get the benefit or am

40:48 i going to get them your network will get the benefit so because we don't want to make a successful authentication with your

40:54 organization where you are predominantly using that device okay so organization specific organization specific

41:01 otherwise there'll be some security implications that's sort of where i was going yeah no it's organization specific

41:07 yeah so the key part here again was we already have the roaming data that you may have seen on the client insights

41:13 from marvelous queries about how is the client home experience we are now getting higher efficacy even more

41:18 granularity because we are able to now merge that with the data set coming from the client android and windows we see

41:25 this as a huge thing especially in higher ed with a lot of windows devices even in the enterprise space we keep

41:31 hearing our customers say hey can you please tell me when you need to update the driver on my corporate devices

41:36 exactly i see the rolling eyes and we're like we have to solve that problem now we can

41:42 and again it's all about third party data it's any device out there any windows device windows 10

41:47 any android device we have the information yes keith you mentioned to answer sam's question

41:53 about the organization specific this data you're collecting from organization a could be very beneficial

42:00 organization b even though they don't have the clients is there any mechanism to to have that learning take place across

42:07 the entire mist from an anonymization absolutely so once we learn certain device types certain drivers we can then

42:14 make a recommendation and that's this is the start of that journey to say how can i give my customers benefit of what i've

42:20 learned in certain org so if certain os's certain drivers are not behaving properly that could be a proactive

42:25 recommendation from marvel saying this just be aware this is what we've seen so absolutely but anonymously and

42:31 and and keith um what we do today already when it comes to you know cross organization is you know we are studying

42:39 and learning devices by device types and there are rf behavior or ssis you know

42:44 at water rsi what mcs rates hold all this kind of stuff and when we are doing this predictive notion of when a you you

42:51 know how much performance can i get from a user at a given hour society given you know you know set of vectors that

42:58 learning we anonymize and learn um you know across device types across organizations we know when iphone 10

43:05 behaves at given the ambient rf parameters right so there is the the mechanism exists more

43:12 of it could be done when we add the marvis line to it i would say a good example is this you know apple released

43:18 an os version that broke all the portal pages when you downloaded large

43:23 large disclaimer things that that's type of global information that okay guys heads up everyone

43:29 you know we know there's a problem uh we have disability across all android i mean we know which we know how much

43:35 better android apple is versus android on ble right and bob we do this for location anyway right right use a global

43:40 machine learn value across all words for devices so yes so well is there a special license for each device that you

43:48 have to add or is it just with one specific marvis if you have marvis you have this

43:54 if you don't have marvis let's talk to see how we get you marvis so you would highlight the driver issues

44:01 with the the windows sdk is there also an integration that shows or does it highlight

44:07 that that driver problem is having a problem with specific features like this

44:12 driver works with networks that have or doesn't work with networks that have k enabled but does with it disabled

44:19 we are yeah i would say in that learning journey we want to start getting the data first before we get to the analysis but that's a good point is what i would

44:25 say yeah i would say to some extent that's kind of built into our wire mutual information yeah so there are

44:31 cases where like zebra devices where we have basically isolated that there's a connectivity problem and we get it down

44:36 to a particular os you know we that's that's even feature which we will then evolve

44:42 to all right so now um so we talked about the windows client we talked about the ai audit log

44:50 innovation as part of our marvis journey let's get on to the second category that you saw in terms of

44:57 new marvelous enhancements the enhancements we've done in line with our ex 4100 what we call

45:04 experience the unexpected launch is all about highlighting wired issues

45:09 that do impact wireless users right now every time i talk about wired problems i see people like just zoning out they're

45:16 like it's a switch it's an access it's a just a power source how does it matter trust me my friends wired issues happen

45:23 all the time you just don't know about it but they show up as wireless issues and that's when you're like okay what do

45:28 i do now packet captures it's not a wireless problem to your point fingers go like this right so what was our goal

45:35 here our goal with morris again was if with a general per stack i get a lot

45:40 more information i can give you a lot more wired actions if it's not a juniper wired stack it's a

45:45 third-party stack i can still give you some indication of what's the problem that again is impacting a wireless user

45:52 so let's go into what we call marvis wired actions some of this you've heard before you all heard about bad cables

45:59 you all heard about missing vlans those are common to both juniper networks and third-party

46:06 switches so when you're running a heterogeneous network mars is still your friend to tell you hey this set of

46:12 wireless users is being impacted because of these wired issues in a third-party wired network but if

46:19 you have a full juniper stack now we can tell you port flapping issues now we can

46:25 highlight where the loop is starting do loop still happen in the wired network no right you would think

46:31 show up hands never there you go [Laughter] so exactly right so and we have been

46:38 like saying we have all this data we should be able to help network operators

46:44 identify proactively hey there's a loop in the network and is starting at this

46:49 port we won't take the action but the first step that we've taken in that journey is to identify where the loop is

46:55 starting because that is what takes hours and hours and sudhir can himself vouch for in certain cases days to

47:02 figure out what is going on so again in terms of marvelous actions right so this is our juniper network sunnyvale

47:08 campus you can see all the switches there the key point again take it back to user experience what we are looking

47:14 at not just the thousands of switches you see here what we are getting telemetry from is every port every few

47:22 minutes as well as asynchronous events so now using that telemetry information

47:28 we're able to give you pre-connection post connection metrics even on the wired side whether it's a wired only

47:34 user or it's the wireless user at the end of a juniper missed access point

47:39 plugged into a juniper switch but is telemetry coming from 50 000 ports not

47:44 1000 switches that's that's the first point right again the focus is on the user

47:51 with that telemetry now what can we do so what you'll see here essentially is

47:58 our ability to ingest that data and then we'll go into what we do with

48:05 this telemetry information coming into the miss cloud so you have all these switches connected

48:10 you have a thousand switzer just in the sunnyvale campus you have 50 000 live hot ports and now let's get into what

48:17 has marvis found proactively and creating tickets for the juniper idea team right now when they find issues so

48:24 all of these mars actions you see are actually happening on a live network where sojay has just gone to present

48:30 rqbr uh to rami our ceo so the first thing is again you're all

48:36 familiar with bad cable so i won't gloss over it but this is something that nobody ever thinks is a problem

48:42 and i can't tell you how many times you've just gone into a poc marvelous within a couple of hours full of the bad cable and the customer goes

48:49 you know what with the previous vendor users were complaining on the wireless network and i just didn't know what the

48:55 problem was it was as simple allegedly simpler thing as a bad cable but now we

49:00 have the proof if marvin says it's a bad cable take it to the bank it's a bad cable just send the tech out replace the

49:06 cable and you're done right but in addition to bad cable and abi thank you for joining i would invite you

49:11 here to the stage um abi is one of the the key reasons why we have so many learnings on mars for

49:18 the wired network so we thank you for your contribution but in addition to bad cable the next one is sport flap now does ford flap

49:25 happen in any of your wired networks all the time thank you it could be client driven right but the whole goal

49:32 for us again is using marvis using aiml in the back end to baseline every port if it's a typical trend where the

49:38 portfolio certain are the day certain day of the week because that's just when the client authenticates ignore it but if it's anomalous port

49:46 flapping what we will do mars will proactively fall out exactly what port is flapping and give the administrator

49:53 again to your pointer we're going to automatically disable address the admin in this case we asked the admin to say hey

49:59 you have the option to disable the port because we are telling you this is anomalous this is not part of the typical baseline trend anything you had

50:06 to add on this one most times these are devices which are misconfigured probably

50:13 uh um probably a card reader that's sitting there which probably needs a ten half but you can figure it as auto and

50:19 then you need uh it probably is continuously looping and these are elements that we can find out these are

50:25 headless devices that don't talk we tell you this this is where the problems are now let's get to our favorite loop

50:30 detected like we all agreed these happen all the time and it's always to say the least a hair pulling exercise

50:37 to figure out where the loop is happening so we are early on in the journey follow protection what we have

50:43 been able to get to is a high efficacy of detecting where the loop started no

50:48 action yet it's more like a driver assist mode to let the admin know so that they are not working with thousands

50:54 of other people trying to figure out every port that we switch and try to isolate the issue we help isolate the

51:00 origin of the issue in driver assist mode as part of our aim and mrs action framework but would you

51:06 like the admin to take that next step of what to do once we highlight the source

51:12 anything else to add to this um yeah it's it's a it's a good starting point uh there's a lot more that will

51:18 come through as a part of the loop detection trying to identify what port that went up what commit that caused the

51:23 issue things of that nature now the even more critical whenever an issue happens on a wireless

51:30 user or a wired user it's always a network problem right it's not a client problem can never be a client problem

51:37 we know that's not true whether it's a driver issue on the wireless side or it's a back cert on a wired device or a

51:43 wireless device how do we qualify that how do we identify that how do we then notify you proactively so we have been

51:50 doing persistently failing client identification for the wireless users

51:56 earlier this year we have now added that same capability on the wired side as well

52:02 so if a wired client comes on and the search is expired

52:07 we will notify you because marvis knows marvis can see this client is in distress my recipe the scope of other

52:14 clients connecting to the network wider wirelessly look at the switchboards look at the aps and say no it's not the ap

52:21 it's not the switch it's not these are the vlans it's this particular client a set of clients and highlight it is a

52:28 client specific issue on the wired side and due to what reason

52:34 uh one of the side effects of a persistently trained client uh a client that has a bad cert or probably is does

52:40 not have the mac address in your database is it consistent continuously is going to hammer your radius server

52:45 you want to know about these clients again at least devices that don't complain you want to be able to go and

52:51 look at amongst all of your you know gazillions of sites which and marvelous actions has always been arc based right

52:58 amongst your in this example the juniper networks we thought we spoke fifty thousand ports we we we are speaking

53:05 about those uh six persistently trading clients and in our customers the scope just

53:11 continues to increase because of the size of the scale that we have so very very helpful uh in terms of reducing the

53:17 impact on your radio servers and also identifying which which devices that can't speak cannot open tickets to be

53:22 identified and last but not least go ahead in the troubleshooting of

53:28 cables uh do you also troubleshoot fiber or potential bad sfps optics we do uh we actually

53:36 send out alerting for bad optics juniper inherently has an ability to identify

53:41 bad optics and and send out alerting as well absolutely present so the question was those are in here

53:47 do we also as part of bad cable identify bad optics the answer is yes we do so i do want to get to the last

53:53 enhancements on marvis as of today tomorrow maybe a different day another new innovation you saying we'll come up

53:58 with thank you but the last pieces we talked about the client-side

54:04 issue of figuring out dhcp problems authentication problems we've also

54:09 expanded our authentication action from wireless to wired so now if it's a dhcp or dot one x problem on

54:16 the server side we have added the switch scope the vlan scope so we can tell you exactly

54:22 which vran is getting impacted therefore all users on that vlan are getting impacted or which clients on a

54:28 particular switch are having a problem because of what dhcp server or what radius server is not responding to

54:33 requests so it's again a client-side view and a server side view with switches and accuracy points so you know

54:40 again the exact funnel of a certain device connecting to a network and where exactly is the blockage happening so no

54:47 more again finger pointing mars has data from both sides it puts it together learns it and gives you a ticket to to

54:55 look at but not just say hey you have a problem tell you where exactly the problem is so you know exactly where to

55:00 escalate to the dhcp team the dot connect team or the mobile device team

55:06 and with that we go on to our our next slide so the next enhancement was you've all

55:13 heard about conversational assistance you all heard about and you've all seen

55:18 the bob's unhappy why is that use case mars has come or the marvelous

55:24 conversation assistant has come a long way since then right we started our journey in wireless we then went to

55:31 wired last year and earlier this year we've been busy with van and then you just saw g-shang talk about the

55:37 third-party data ingestion where we can now look at isp reachability as a journey to looking at the application

55:43 experience great information but how do we make this easy to consume for a level one

55:51 help that's when that first trouble ticket comes in do they need to go look at sle do they need to go escalate every

55:57 wireless ticket to attack three show of hands you guys tell me when a wireless ticket comes in today

56:03 how much of those wireless tickets are escalations to tag three ten percent twenty percent or almost

56:10 hundred percent one hundred we say it depends

56:16 you mean missed at zero everybody else is 100 you can say that sam that's fine

56:21 no but honestly like our experience with working with our customers always been that every wireless ticket by default the

56:27 ticket gets opened either no action is taken or it gets escalated because wireless wireless i don't know what's happening right so that's the first

56:33 problem we tried to solve with with marvis conversation system but we know

56:39 even if it's wireless is not the culprit for an end user it could be wired it could be even it could

56:45 be isp could be application so what are we doing with the marvelous conversational

56:52 assistant to make the help desk experience not only better

56:57 but more impactful because end of the day what happens the faster they're able to root cause the issue proactively the

57:04 faster your experience is restored back to normal and that's what we care about that end user experience right

57:11 so in this view and i'm going to go through it a little bit fast you're essentially going to see how we start the conversation with marvis and try

57:18 this out in in your networks if you have a missed account this is a production network you can now go in and ask

57:25 marvish to double shoot a wireless user you can ask marvish to troubleshoot

57:31 a wired switch you can ask marvel to troubleshoot a wired client and i'll tell you exactly for every domain

57:39 what is the problem right so here i looked at wireless it's telling me slow association it's telling me low power

57:44 it's telling me limited capacity and then rml kicking and solve those problems now i'm even saying hey go

57:50 troubleshoot the switch for me marvis because i'm getting some wired issues being reported to me so imagine your

57:55 level one helpless user does not need any training any certification and no

58:01 networks 404 not just 101 but is able to get the answers literally at their fingertips of what's impacting the end

58:08 user experience so here in the wire on the switch side you can see exactly what it's showing you

58:14 it talks about troubleshooting switches you can now see it's also going towards the van edges so if you all you have is

58:22 the wired network or if all you have is the wan network i often get this question asked that hey stinolini rs is

58:28 great but what if i don't have the full stack what if i only have wireless or what if i only have wired or what if i

58:34 only have vans if marv is still useful absolutely yes yes question any plan to have marvelous

58:41 conventional assistant in other language in english and yes absolutely so we you will hear

58:48 from us later this year and how we are localizing the dashboard itself in other languages and then following that will

58:55 also become the morris road map so we talked about how it's not you know the one thing i want to say is that we

59:01 were having the discussion yesterday with bob as to how should we talk about this because whenever you say ci you'll

59:07 have every other other vendor talk about chatbots right what does a chatbot do a chat bot

59:12 answers simple questions documentation or it'll point you to here's a list of faqs choose your answer

59:19 right it's not truly conversational it has zero context on your problems

59:25 and this is where the marvelous conversational assistant and we truly call it an assistant versus

59:31 just a chatbot it's looking it's truly the face of the marvelous ai engine it's

59:37 enabling to answer as bob says very complex questions

59:42 in a very simple manner but there's a lot of work happening in the back end right all the baselining all the machine

59:49 learning all the anomaly analysis all of that is being concisely presented as oh

59:55 you have a problem on this end and this is a reactive workflow here the reason why it's happening here's a

1:00:00 recommendation and also here's the scope so it's giving you the evidence it's giving you the the scope of impact and

1:00:07 it's telling you exactly what to do if we can't fix it for you what to do to fix it so it's the full

1:00:13 feedback loop of a question and answer with a recommendation across the full stack

1:00:19 wireless wired ram how can you tell when in your conversation um how do you give negative

1:00:26 feedback to the conversational engine because if i ask i mean i've asked several times hey show me interference

1:00:32 and you're the ceo and and the answer is not what i'm looking for how do you identify that

1:00:37 that's not correct so you fix it great point right so as bob said marvis is not dead at the 100 mark it's probably the

1:00:44 75 to 80 percent mark but after every mars square we do ask you for a thumbs up thumbs down and we ask the user to

1:00:50 give us input that's the input to take back saying exactly that and that's why that's critical i know it's annoying

1:00:56 sometimes but that is what we used to train the engine saying hey you had the wrong answer you may have had the data

1:01:01 but the answer was wrong or maybe we didn't have the data so we then go back and train the model and pull the right data

1:01:14 you