RIPE 90
Plenary session
Main hall
4 p.m.
BRIAN NISBET: Okay. Hello. Could I ask you all to take your seats please. Okay, so, my name is Brian Nisbet and along with Doris Hauser, we shall be sharing this second Plenary session. So we have a couple of full length talks and then we have our first lightning talk session in this as well. A quick reminder, please rate these talks, think about them, give us feedback so we can make even more amazing programmes for time to come.
So, without further ado, I shall invite Nina to the stage to talk about the routviews project update.
NINA BARGISEN: Hello, hello. Hi, I am Nina, I was just realising I actually never spoke in a Plenary at RIPE, which is kind of interesting, I spoke at many other places but that's fine, this is my first. Be nice.
Today, I am going to talk about the routviews project. For those of you from the British Isles, you would probably call it routviews project, but it is an American based project, so I will be calling it routviews, so, when I say route, think route if you are from those areas.
So, why are we here and why are we talking about this project?
Because, if you can read all of this text that we put on this slide, this is a very old project that started back in 1995, but it has been around for a very long time, and we must admitted it probably also had been forgotten a little bit by some people and there might be some new folks who don't know about it, so this is why we first of all trying to give it a bit of a restart but also going and telling people about what is it that we do, why are we here? Why are we here along with also the very brilliant probably here in the RIPE region, which is the routing information system that is run by the RIPE NCC? And who are we? And what do we do and what do we plan to do?
.
So that is going to be the topic for the next 20, 25 minutes.
So, we started in 1995. We now have around, it says 40, that actually we have now reached 50, collectors worldwide. We put them into Internet exchanges and we collect routes. Just as most of you will know that the routing information system do, and some of you with grey hair, just as me, will remember that we also do.
We collaborate with CAIDA and we are supported by NFS grants in the US.
It is now based at University of Oregon and it is now operated by the network start‑up resource centre. It used to be its own project when it started at the University of Oregon, but over time, people have retired and at some point NSRC was asked to take over and help both with funding, but also with the technical ‑‑ with running the project and building it out. And that is what we're doing, and they are hired in a new team.
This is the team here. Hans is where here in the room. Me, Phillip Smith, Owen Conway, so some of you might remember, Anton, who is also somewhere in the room, and Philip. We are the current sort of base members but we get support from many other people as well.
And we are now trying to sort of move the project along.
So, what is it that I'm talking about? Well I mentioned the routing information system, assuming all of, you know, but we're going through a little bit of a recap maybe of this morning's BGP tool. BGP training session but also a little bit about exactly what it is that we do.
So, basically, it's a tool that allows Internet operators to look at the BGP table from different backbones and different viewpoints throughout the world. And we have many use cases we can use that for and we put them into Internet exchanges.
It actually turns out, and when I joined the team I was actually surprised to learn just how many of the tools that I have been using in my 20 years plus peering coordinator career is based on data from both routviews and RIS and other BGP collecting. But we have CAIDA AS rank and BGP reader is a tool that you can use to read our data. Hurricane Electric BGP tools kentic market intelligence, BGP monitoring, catch point, many of the tools that a lot of us use many days will pull in at some point data from our system along with other data that they use.
And we try to spread the collectors throughout the world. It is, or has been a little bit focussed on US and Europe in the beginning, but we are now very actively working on getting more collectors out in the sort of further areas of the Internet. Naming Africa, southeast Asia and Latin America.
So, just to give you a little bit of a recap of the BGP workshop this morning. For those of you who didn't make it, so what can we actually read out of the data when we look into the BGP routing table?
.
I know a lot of people in this room will be very familiar with it, but let's have a look anyway.
First of all, we have two different, we have two very ‑‑ two groups of users, I think we can say, of the data. One is the researchers. The researchers are using a lot of the stored data because what I forgot to mention before is that we have been around since 1995, we have been collecting global routing table since 1995 and stored it, so it's still there, you can go in and you can have a look.
So, researchers can use this to go back in time and see what happened or how has everything developed, or whatever they want to use that data for.
We also provide a live stream of BGP updates which means that you can use the data as well to go what is happening right now in near realtime.
So, this means that we basically have researchers and operators as the two main user groups.
So, me, coming from the peering community, my favourite use case in using routviews data before I had access to tools I would have to pay for would be to go and find the data I need to find to do a good peering negotiation. So basically, look at the global table to see who are the upstreams, who are the peers and who are the customers. And it's all about the AS path and where do we get a lot of data collected with AS paths to look at is to go to the multihop collectors we have.
So, we have two types of collectors. Those we put on Internet exchanges and then we have a set, a number, I think we're at eight collectors now at University of Oregon, where we do multihop sessions primarily with operators who do not have a presence at an Internet Exchange where we are.
We do require to have full routeview. So we want all of your routes when you peer with the multihop collector, I'll talk more about that later. Going to a multihop collector you will go to a collector with a lot of full tables on so that is a good place to go look and start playing with your BGP reg outs to figure out how is this ASN you want to analyse connected to anybody else.
So, here we will just be looking at the prefixes originated in a particular AS. I just randomly picked out on the Internet. Let me see if this works.
So we have the reg outs up here and we have, we can see the originator routes and then we can see the next ASN. As we all know, BGP, this is the next hop, or this is the connected ASN to this AS.
Here we have, we see that the originating AS is prepending, for me that is usually a sign that this is a transit connection because we don't really want traffic to flow that way if we can avoid it. So, let's put an extra ASN in the AS path.
And we can also just look at, we see that here, the well known Tier 1s in the AS path and we know that here we are also looking at transit relationships.
So, I can note this down in my what do I want to know about this ASN.
The other thing that could be interesting to look at is who are the down streams. And those are usually easier to look at the local collector but they will also show up in the multihop collector, and here you just use a different reg outs ‑‑ maybe this is a class on reg outs. Anybody want to have three hours on how we do those? No? Ok. So we have, we have it here, this is what the ASN announces and we want to make sure there is something after as well. Another originating or maybe even longer. So, we get the list of customers by looking at the ASNs that show up in this.
So, that way you get upstreams, you get customers and you get potential peers, and then you know a little bit more about the network when you go out and want to talk to them about why they don't want to connect to you. That was basically that use case.
I have one more use case to go through. This is what I call make life easier for your NOG. Because often when you get a call with a routing issue, newer NOG will go and maybe look in their own routing tables and try to work out what is going on, and the, one of the original ideas of starting the routviews project was for NOGs to have access to look at how does my routes look from somewhere else on the Internet.
So, here we have an example of somebody is trying to balance traffic between the upstreams and then for some reason the deaggregation doesn't really work or when the deaggregated connection goes down, things are blackholed, something is not working well and this is not great.
.
So, we start at looking at the covering prefix, and we are paying a lot of attention to the RPKI state, and we see that oh, the covering prefix is ‑‑ no, one of the longer prefixes has an invalid RPKI state. Oh no, this is not good and if we look at the shorter prefix, those are will valid prefixes. No, validation D oh my God, I am so sorry, I feel like I am jet lagged because out of solidarity I got up very early this morning to get here. I could be with my American friends with jetlag and we would feel the same so apologies for mumbling around. And apologies for actually turning this use case the wrong side around, because the point here is that the longer prefix does not have the right ROA so we forgot to update that, and this is why our deaggregation scheme is not working. And we can find that information by looking at what's going on on one of our collectors.
So, we'll quickly go over the conclusion, because that was wrong.
Okay. So how do we get all the data? Well we get the data by peering. So, some of you will already be peering with us, but we are peering and we used to have an open peering policy. So we would peer with everyone and their mother, just they asked us, we would go yes, peer with us. But just has been the case in the project that is run here by the excellent people at the RIPE NCC, we have found that we need to balance the resources that we have between getting as much data as possible, storing as much data as possible with the operational cost of both the storage but also the capacity on the collectors. So, we have introduced a route routing policy so we no longer have an Open Policy but a more selective policy.
So, who do we care about peering with and what do we want to do?
.
So, we still want to collect as much as possible about how does the Internet look today. And what has happened over the careers since the beginning of this project where you could probably see a lot of how the Internet was working and who was connected by who, by peering with the relatively big backbone providers and a selected few of the regional providers, would you get a lot of information about where is the main flows of traffic flowing on the Internet, but this is not the case today.
As we talked about, at least in the peering community in over many, many, many years, the Internet today is not very hierarchical but it has become a very flat Internet, where a lot of the content has moved to the edge and is served on the edge directly to the consumers, very short AS paths between the content networks and the consumer networks.
So in order to catch that, we really want to peer with a lot of people who have good routing tables but they might not be a global table but it might be the regional table or it might be a local table from an IXP.
So, we want to make sure that we have capacity to store that route and to have collectors at the right locations throughout the world to collect that. And this means that we have to free up space on some of the collectors, and that means there are certain networks that we really don't want to peer with anymore. The
.
So, some of those are what we call hobby networks, and it's a difficult decision to make because we actually do love people who are doing BGP research also from their home or from their own little private network that we are building up or their experimental network or their student network, but we also find that we have a lot of operational overheads from these networks on our system, we also find that maybe they are not showing the right picture of what the operational internet is looking like where the traffic is actually flowing so we have decided that those are some of the networks we will say no to when they ask for peering. We still love those people, and their routes will make it into our system through their upstreams or the people they connect with and have success in peering with so they will still be able to do research and, like everybody else, they will have access to our data, so we love BGP researchers, but we are not going to peer with every single one of them on earth.
What we also do is when we want to spend resources on having a bilateral BGP session to a network, we would like to see a full view of the Internet. We want as much of that are routing table as they are able to give us from that location: If they only want to give us their usual peering routes, well then we will likely see them on a route server and then we're not interested in also spending time and resources on a bilateral peering.
And of course, as a peering coordinator for routviews, I will always have the final say so if somebody is doing very, very it go stuff, we might just overrule all of this and then agree to peer with them any ways because it's an important information to have into our system. And by the way, if anybody has questions, just go to the mic and wave at me and say Nina, also if I'm doing something really weird.
.
So what are we working on on the back‑end?
.
So, first a little bit. We're using the daemon we're using BGP using is FRR and I think we have everybody updated to version 10 now, except for the legacy Cisco that still uses Quagga. Another project we have is that we want to get away from actually installing hardware at physical locations throughout the world and just instead of asking for space and power, we would be asking hey, can you give us a VM or something similar? Or we might, we have also been talking about where we put hardware out we will create VMs and maybe offer VMs to other people as well.
We are constantly building out. Not very fast, but we do feel we are not quite yet where we need to be, and we have a prioritised list of Internet exchanges or Internet hubs where we want to be, those include more locations in Asia, more locations in Africa, also more locations in Europe and South America.
Recently we have added Romania and Italy, so we are still slowly working on moving on.
Another thing which is important that we are working on is that pretty soon we're going to launch LookingGlass, so instead of logging in on the collectors, which is the most common way of looking at the data today, if you are not downloading a whole bunch of data, you can go and use Telnet, you all remember that protocol ‑‑ so you can use Telnet and log into the collector and then just use your usual BGP routing command. We want to get away from that because people are also ‑‑ some of our users are using it programmically so we have scripts that are constantly logging into the servers and doing BGP commands and that is actually putting a strain on the servers.
So, we will do two things: One is, launch a LookingGlass front that will still give you access to the individual collectors but through a LookingGlass front. And the second one is get people who are programmically pulling data from the collectors, moving over to the API instead. Like everybody else is doing in this industry, so we want to do that too.
There is a beta version of the API, but eventually it will be having all of the collectors available.
And then eventually, we will turn down Telnet access except for a few collectors where we will still remain to have that, just for fun.
All right.
Otherwise, we're working on just updating our hardware and the storage and infrastructure because some of it has been around for a long time. So it needs a revamp, and we're working on making life easier for ourselves, just like everybody else, by automating a peering configuration, automating managing the whole infrastructure so we can keep doing that for a growing number of collectors and still remain the small team just like everybody else in the industry.
And at the same time, we have a couple of people also working with FRR, testing it, making sure, report bugs back and making sure that it develops to become a good and stable BGP routing daemon.
All right.
And I am reaching the end I can see, so what we want to do in the future again is more collectors, scale our archiving, maybe diversify locations and make sure that we talk to the community and the users, which means people in this room, who will use, or are using our data and learning what do you want us to do to make sure that we keep supporting the Internet community as we have done since 1995.
We also are very, very happy for the people who support us, I talked about that we would support it by NFS grant in the US. That, along with a number of private and public sponsors as well, and we say thank you very much.
And we really, really appreciate any kind of support.
Other ways of supporting is help us host collectors throughout the world at Internet exchanges, and of course for you who have interesting routing tables that we want to look at, come talk to me about peering.
And that was it from me. And I will now take questions.
(Applause)
BRIAN NISBET: Do we have any questions? Is do we have any people who suddenly want to volunteer to be hosts?
NINA BARGISEN: Otherwise I need a poll, who knew about routviews before I did this talk? Who likes routviews? I am preaching to the choir. Thank you very much for having me any ways.
BRIAN NISBET: Okay Nina, thank you very much.
.
BRIAN NISBET: Now I am very much hoping that Chrissey and Tobias are in the room, at least one of you is.
TOBIAS FIEBIG: Even though there is a joint work together with my student Christy, I'll be here alone, because Christy had other obligations. You all probably know me from fun talks about worms and routing cans, networking, getting rid of v4 addresses on transit, but there is also the stuff I do to actually earn a living which is being also kind of a bit like a security researcher. And this is what I do there.
So, finding funny things on the Internet. So, OpenSSH, most of you know it I guess. Who is still using Telnet?
.
Be by the way RSH is also kind of like not a new fun thing. But it doesn't really matter if you use SSH or not because it has a tendency of inviting guests. People that feel really invited by you considering route a good password or password or your users doing so. They then come and I mean like guests, guests usually bring presents, and presents are fun. At least I usually enjoy when my guests bring presents. We do here have one of such examples of a guest coming in, apparently, MDRFCKR, it's a name they gave themselves, but they came in, left a little present and that little present is a bit interesting. Because that little present is an SSH public key and they are amazing, they are kind of easy ways to get rid of passwords, easy ways to log in. And SSH has this night authentication protocol to use these keys. And beyond all the technical parts, what is important to remember is if we have SSH public key authentication, we do SS client, we say hey following user, following public key, I would like to log in. And then the server goes to the user account, looks for the key, it find the key, it's like hey, I can build your challenge, I send you the challenge, and then you go go about it. Anybody notice anything? The interesting part is that the server looks into the account and it also sends a challenge if the key is there. Which kind of makes sense because otherwise you could overwhelm a server by making it a lot of public key crypto which is expensive by starting a lot of requests and making it do this challenge stuff. But it also means that we are getting a signal. Because if we do not get back a challenge, the key is probably not there. That is actually been known for like a very long time. For some reason people filed a CVE for that. I would call it like a feature. But ‑‑ well, people.
We tried this. So we build a little Zgrab2 module and we tested it on OpenSSH and these and it turns out it does work, if you have a public key like the people figured out over five years ago, you can go to a server and be like hey, do you have that key installed?
.
And it works, but what do you do with this knowledge? If you want to have some insights, let's call it insights.
Well you have to scan for something, and here we are very grateful for to bit defender who actually shared round about 50 really funny public keys among those are friends with the interesting M starting name with us, and this is an overview of these keys and at the bottom you see a timeline. These lines correspond to how often events in relation to these keys were seen during the defenders incident investigations. So, kind of busy beavers that left their SSH keys unattended, public keys that is. And important: We are only getting public keys. So even if we would figure out that these keys were installed somewhere we could never lock into that server because we do not have a private key. They do have that and they probably already locked into the server, but we can't, even if we figure out such a public key is installed somewhere.
So, what did we do? So, we scanned the whole Internet, a great way to make friends, for open ports that are related to SSH like 22 and 2222. Like you can scan more than one port so if you one your SSH on 2222 you might get rid of some of the noise, but you are still being found. We then actually tried to log in first with a freshly generated key, because that allows us to determine if the server is A, a honeypot or B) has a very funny implementation that also turns a challenge even though it shouldn't. And then we go through our funny keys.
Also, I know this might come as a shock to you that there are people like that out there, but we actually did read our abuse@ and we also replied.
So, we went over the whole Internet, we did that for three users, route, admin and U database, which apparently is like from some embedded stuff very popular thing, and we had in the end 52 malicious keys because we felt the need to include two keys we had recovered from GitHub, and you also know that GitHub is sharing all our public keys, right, if you have the user name you can retrieve the public keys, really useful, so we also retrieved that for like freshly departed contributor of exact. We didn't find any of those keys. And then we hit the whole Internet for IPv4 and IPv6. And we are still doing that. So we are continuously scanning so when we're done we're starting over from the beginning because we are sharing that data with foundation and the Certbund so the federal CSIRT in Germany.
And yeah, it kind of sticks. As you might see there, our friends with key ID 6 was a funny name. They are actually quite active on the Internet. So we got several thousands hits just for them, in total across all keys we got around 22,000 hits, and what we hit included things like critical infrastructure, you projects or something something database, so interesting stuff, and there was also one case, I don't know what kind of device it was, but it was like a maximum of one hop removed from the border, so if you would go through an X it would be one hop away and TCP port 179 was open. I'm not sure what kind of device it was, but we better told the people who were actually allowed to log in that they might want to check out whether there is something on there which doesn't really belong there.
Anyway, this is interesting. However, specialisation is also important. We know that from all our business trainings. And that's also something we see because some of the keys we scan for actually belong to people that are more into IoT. I mean we have an IoT Working Group at RIPE. Apparently they also have been IoT Working Group at malicious actors whatever the organisation for that, I am pretty sure it's also membership based.
Anyway, most of them though, of course, hit OpenSSH because most of the SSH daemon is publicly reachable on the Internet are OpenSSH.
And also, as expected, we did find compromised machines all over the world. You are currently seeing China kind of lighting out, but there is also a lot of IP addresses in China. I mean the US are also relatively light, right? More IPv4 and IPv6 addresses, well more hosts, more hosts, more compromised hosts
The thing is if we look at the organisations it usually looks at kind of expectable. Theres a couple of outliers, when we used data feed and I think we used the BGP tools feed for that, there were a couple of inaccuracies which is also related to that flattening of the Internet because sometimes it's really hard to distinguish what is Ali Baba Cloud and what is Ali Baba VM hosting. Interesting for friends that was for a large hoster. That was classified as a CDN and that's why the CDN for France is so high.
What we also got were anonymous love letters, so people tend to not appreciate it if you are starting authentication requests with SSH daemons. Because in the server logs it kind of looks like normal brute force, it's an authentication attempt that started. We could never log in but it kind of looks the same.
And there was some people who are really pushing it. I was always a bit confused because they were pushing it like really heavily, and I was wondering are they also doing that when we are writing an e‑mail to like this really unknown middle of nowhere somewhere between Paris and Tokyo, located hoster that just sends app, use app at def now, but they were really pushing it right. Wanted to drag us in front of court, etc., etc. Etc. These things get really, really fun if based on the domain, you actually know the person. It changes the whole tone of the conversation. However, they were not the only ones writing funny letters that make me like oh, it might be kind of getting a sixth sense here..
Because we also got these. Somebody finds something funny about this because this is like app use messages for R v6 scanning. When we looked at the amount of app use messages we got a lot more engaged messages when we were hitting v6 and we were also getting these. Like, there is more of them, right. This one is in German, it's from I think an energy provider, but same pattern, there is this percentage sign and the underscore lock if, somebody recognise that? Like, this might be a bit more explicit, it's a Linux system though. And like, when we checked, like we did not get this kind of messages for v4, we're like oh, what could be an edge 01, edge 04, edge 05? And I mean what is an NW core?
.
Well it turns out for some reason we got a lot more app use messages when we did v6 scanning from people running routers, running bit routers which have lock messages which surprisingly follow the same formatting you usually see when you have a Cisco or a Juniper. So, I kind of think that this is not the peeling CPEs which found their ways into the Internet. And this one here, is actually from a Tier 1, whose peer routers we apparently hit. Please, please, please firewall your loop back also for v6 not only for v4, you are apparently doing it for v4, otherwise we would have got more app use messages, but also do it for v6. It's like yes, we might currently be the only ones doing this, but there is also the others and we do not want to have more hits on v6, especially there.
There was another thing, and this was actually quite odd, because we were browsing through the data and like I picked up on this key, like it had this huge histogram of different keys and like what ASes they were founder and I picked up on this key because it was basically found in two countries, well regions, some European countries, and middle eastern country with tense relationships with many parts of Europe, and in Europe, it was basically two ASes that had hits for this key. Like I recognised the AS numbers because they were both prominent service provider that was a very affordable service offering focusing on the commodity market in dedicated servers in virtual machines. I was like that's odd.
And what you also see is that most of these hits, like total hits, are actually for the same OpenSSH version. So, these hosts in the middle eastern country and those hosts running in affordable hosting companies in western Europe, they were all running the same Ubuntu 2004, which is kind of odd, because usually with these kind of compromised hosts we would see this Haribo mixed bag of things where you would get like all kind of different SSH versions because these systems are like, they are compromised, they are not like centrally managed by the attackers. This was a little bit odd. We didn't really have an explanation. But when we thought about it, we figured that this is a problem for other people, just let other people solve it until other people told us like, we took a look and you can now like say that you found M key 48, which by the way just our internal designation, we ordered just the keys which frequency. Like by frequency in the input data.
We can also look at this one funny host offering very affordable services because they were actually on our opt out list. So, we were kind of not allowed to scan them because previously they had complained about our scans in other matter so they opted out, so I did the most reasonable thing one does in that situation; I annoyed the hell out of them until they said sure, if you stop bothering us, just go for it. You stop e‑mailing us then, okay.
They were actually the single biggest hitting AS in our dataset. We would have skipped on that and we would have skipped on MK 48. They have a total of round about 60ish hits for our friends with MK 06, which is like this M funny called group, and then they also have round about the same amount for MK 48 on different hosts. Granted, and of course, we do these scans continuously and then opting back in actually allowed them to do some funny spring cleaning there. So, all those hosts round about 1 big gigabit per host we took away a really nice chunk of toy from the controllers of MK 06 and a couple of other keys. So, if you want to opt out of funny security ASNs, because I understand that they are not that funny and we also hit a couple of things like NAT64s, like public NAT64s, we got the v6 addresses from that which we got from the IPv6 hit list and we are thinking behind one of your devices is the whole Internet, like the v4 legacy Internet. That's annoying, I get that, and I get why you want to opt out then. But if you do that, please try to at least do your own scrubbing on your own network to figure out where these compromised machines are because they will not only bite you they will do what you find annoying to other people on the Internet and that's not funny.
So, what we also did, is we work with a shadow server foundation to actually get notifications out there, and quick show of hands because it worked so well in the previous talk. Who here gets the shadow server feeds?
.
You know, what this is. This is my face of disappointment. Because basically ‑‑ well, on the other hand, if I think about it, no actually it's not my face of disappointment because I work as a security researcher, so you are all making my life a lot, well not necessarily easier, but you are making me work and I can keep doing my job, but in the end you are making like more insecurity on the Internet which is not funny. So, go to the shadow server foundation, just request that feed for your network, you get regular e‑mails with stuff where you want to look at. There is even tooling, right, there is tooling with which you can automate that, the processing, and then be like hey, maybe, maybe we don't want to run that public whatever service, maybe we don't want to have like 1,000 memory cache servers, it really helps the Internet and it makes a lot more free time for a lot of people if people clean up a bit after themselves.
So, now what? Again, subscribe to the Shadowserver feed. Clean up your networks. And please don't write us angry love letters. We kind of know that you only do that if you know that there is not real attackers on the other end. But researchers, just, just try to be a bit nicer and we also try to be a bit nicer. I mean we try, we can't promise but we try, and some scans are good, some are not. But if you opt out, at least make sure you have your own network under control.
Thank you.
Any further questions?
.
(Applause)
BRIAN NISBET: Any questions.
AUDIENCE SPEAKER: Rob Evans, I noticed one of those e‑mails started "Dear AS 680" so, what's your relationship like with DSN cert and is it better or worse now than it was before the scans?
TOBIAS FIEBIG: They are really forewarning us messages and like, the IP box we are using do actually have dedicated use contacts, so most, most of the messages are actually did hit own addresses. Some went to, or like to the Max Planck society in general, which always was a challenge in getting those forwarded in a way you can reply to. By the way, if you have a cert, figure out how you can forward message and end boxes because people can then reply to them f you make your cert use any form of exchange stuff that will actually harm them in doing their work because then they cannot easily forward those messages in boxes which does not enable potentially responsible parties to actually reply to the requests directly, which might be necessary. And the DFN cert also knows what we are doing and appropriate categorised that and if there is nothing out of the ordinary, they end the request appropriately, especially automated stuff.
BRIAN NISBET: That was going to be my follow‑up question whether you had told them or not.
AUDIENCE SPEAKER: Thank you, awesome work. I wanted to ask you scan the port or three users name route at main end... what does U database stand for.
TOBIAS FIEBIG: U database is I think a name they found ‑‑ like a bit defender found in a couple of engagements. It was like the cert most prominently associated name in the dataset we got. There is a couple of more details in the paper, but it's basically like vendor, vendor, vendor.
AUDIENCE SPEAKER: Leslie Daigle. This is a comment more than a question. I think this has been a great overview of sort of the underbelly of what happens when you scan and what you see. But it's also a little bit of an advertisement for the talk I'm giving tomorrow in first session, where we'll be talking a bit more about our global honey farm which does not scan but is scanned, poked and prodded regularly and sees all of these things and our conclusion is much like your bullet number 2, clean up the networks, but we would also like to suggest that maybe we could do this collaboratively and come up with some ideas of how to do it in an operationally sustainable way to get rid of that back round radiation on the Internet. Thank you for your talk.
TOBIAS FIEBIG: I'd be all in favour of game‑fying it. Most cleaned up network, you get a little trophy at the next RIPE meeting.
BRIAN NISBET: I do have one other question, because obviously when you are making things less good for bad people, they often don't appreciate that kind of thing, and have you had any ‑‑ do you think they have noticed, I suppose is the question?
TOBIAS FIEBIG: A) maybe not. Like maybe they didn't notice. B) most direct threats and harms came from love letters. Like. I didn't have the really nasty ones there which is interesting because those people get more interesting when you answer them like an academic. Often it helps to start insulting them like if I just talk to them like I talk normally to people, it worked a lot better. And the cert thing, it's like well as long as you don't start cleaning up we're not really taking anything away from them, are we?
BRIAN NISBET: A winning combination. Okay, I see no more questions so thank you to you for the talk and your team for the work. Thank you very much.
Applause)
.
Okay. Next talk. Our first lightning talk session. These are no more than ten minute block on a single idea or thing and hopefully of interest to all. First up we have Valerie on starting your own Internet resilience club.
VALERIE AURORA: I learned about something call crisis engineering and a bunch of other things and started Internet resilience club. So I work for bioshock systems, my systems software consultancy. I have about 25 years doing this. I have worked on VPN, TCPIP implementation, network device drivers and lot of other stuff. I wrote the TCP/IP drinking game and moved to Amsterdam from San Francisco in 2023, which is relevant very shortly.
So this is my nightmare scenario. I wake up one morning and I have no Internet or cellular connection. Obviously I have no land line, and all the people who can fix it also have no connection, all of the communications has gone down, and nobody has any emergency plans for while they are trying to fix it, most of them have been outsourced outside of Europe anyway and the network is down so they can't fix anything in Europe and there is Internet backup hardware to fix anything that is wiped or cut or blown up.
This is the new normal for Europe. So, just a few examples, the Russian modem wiper attack on Ukraine, a problem for Europe, but also took out German wind turbines. The NATO head said all of Europe should prepare for war. Amsterdam is currently preparing for weeks without power. Which is interesting given the effects of the ten hour blackout here. And ships can't keep their anchor up when they cross. We have to pretend this is an accident because blah blah blah geopolitics.
What is Europe doing to prepare? Well there is actually this fantastic report from the EC and ENISA about cybersecurity and resilience of the EU's communications infrastructures and network. Great summary and recommendations. It's the implementation that's the problem. How well it's implemented seems to be correlated with how close a country is to current or potential friendly lines. Obviously Ukraine is leading the way on both implementation and knowledge sharing. The reason this talks exists or Internet resilience club at all, is I saw an earlier version of the network resilience talk about the Ukrainian IXP one IX, that is going to be on Tuesday afternoon, you definitely want to go, please go to that, you can watch the whole video as well, it's great.
.
So I live in the Netherlands and I would call what we're doing the opposite of preparing. If the Dutch government does have a plan for restoring the Internet, during the catastrophic loss of communications it's a very well kept secret. I can't find anyone who knows what it is.... Dutch technology firms are moving to US own Cloud, and here's what they did to the Dutch national emergency communication system. Here is what it used to look like, redundant, hardened, wired, could run without main power. Here is what it looks like now. This thing is not going to work. In this situation, we can prevent crisis, we can't prevent crisis but we can use them. So crisis engineering is the study of structural transforming of organisations during crisis that threaten their core functions. Organisations usually avoid risk key or expensive changes when things are working. You probably notice this: This includes everything necessary to prepare for a crisis like I don't know, moving your stuff off clouds that are located over Atlantic cables. But, the positive side is that when a crisis happens, a few people who are prepared with tools and plans can carry out mainly change very quickly. So, never waste a crisis. The crisis engineering was developed while fixing a bunch of broken US Government systems starting with health care .gov. You can learn more at this website. And this will be on the end of the presentation as well.
So I thought what can I do if I can't convince the government or businesses that this is going to be a problem? What if I could set up an ad hoc emergency communications that could bootstrap for emergency exits. Where is the people who knows where there are modems that haven't been wiped, I would like that person to be able to talk to someone else. Everyone would be a volunteer, it would have to be cheap, easy and fun. And we have to be able to communicate for days without any centralised infrastructure. I am assuming the cell phone network, cell tower batteries have run out and there is no power.
I struggled with ham radio, I should be a good person and learn to be ham. No, that's too expensive, time‑consuming and power hungry. The test is in Dutch in the Netherlands. I am only at a 2, give me a break.
Low RA is a long range radios, they can send text messages over a line of sight, they are cheap, lower power, don't need a licence. They use the mesh with the OpenSSH firmware you can send text messages. Simple protocol. They are rediscovering how to network, it's cute. You can attach to it your phone our computer to send or look at the messages, there is multiple channels and there is already about 25 hundred nodes in Europe just registered on mesh map. There is many many more. So this is a website, this is what it looks like in Amsterdam. Here is what it looks like on my phone from the node I have.
So, here is what I did. I invited fun, nice, internety people to hang out. We figured out how to talk to each other normally. Bought some radios, installed them together, picked a channel to chat on and plan meet ups and keep them going and when disaster strikes we can communicate.
The most difficult and annoying part was choosing the hardware, it took me months to research this. So here is my recommendations, this is on a website I can look at. I have got all of these things in my back pack, come find me.
The most important thing is you can actually destroy this hardware if you power it on without an antennae attached or detach it while it's on. Don't do that. Other than that, things are cool.
So this is some of my stuff that yes, that's my folding solar panel. You can start one too. If you work for a network operator, perhaps you could suggest as an annual gift a low RA radio and a mobile phone power bank because people want the power bank, maybe a solar panel, I don't know, we could get really far on having this Internet resilience club.
There are some links, and I'll take questions if anyone has them.
(Applause)
BRIAN NISBET: We have a question here from Edoardo. Which is: What is the dependence on power?
VALERIE AURORA: That's the next thing. It draws a lot on power. The solar panel can recharge it, it's like this big, unfolded.
BRIAN NISBET: You live in Amsterdam, I live in Dublin ‑‑ solar panels are...
VALERIE AURORA: I agree.
AUDIENCE SPEAKER: Speaking for myself. I also happen to be a ham radio operator, so ‑‑ but it's maybe looking at it in a wrong way, like, with ham radio, you can do anything. You can just bounce it on the atmosphere and you are done. But ‑‑
VALERIE AURORA: They are complementary, right?
AUDIENCE SPEAKER: The thing is ham radio is there to test a lot of technology and what is used in ham radio but also in the emergency responses are they are repeaters on hills that are interconnected with their own power generators, UPSes, everything, and something like that with licence frequencies can also be used for other types of communication.
Another thing that also exists in ham radio ‑‑
VALERIE AURORA: If you don't mind getting to the question.
AUDIENCE SPEAKER: Yes, so there is also wi‑fi networks, they dot same that are run by ham radio, that can be done for emergency communication as well, just use RFC.
VALERIE AURORA: Yeah, all of that is true and I am looking for something that's cheaper, easier, simpler than that stuff. But all of it works together and there is overlap and that's important.
BRIAN NISBET: Please.
AUDIENCE SPEAKER: Simon. Short question in that same space: Have you considered wi‑fi halo of this new power thing?
VALERIE AURORA: No.
AUDIENCE SPEAKER: Hello. I happen to be a radio amateur as well. Have you looked at the American ‑‑ well it's amateur radio emergency network, or similar technologies whereby the same thing is done but with very low power wi‑fi.
VALERIE AURORA: Yeah, so there is a bunch of different options one the things I learned when I was researching this, is that many existing emergency communications networks were not working that as well. And that I really wanted something that was available and any random person could just buy a little bit of hardware and start doing it. There is many many options. This one was very easy.
BRIAN NISBET: We're done time wise, apologies. Thank you very much Valerie.
(Applause)
BRIAN NISBET: So, now we have the next talk about the Ivory Tower syndrome operators reflections on academic BGP security solutions. This is part of our RACI programme.
ALEEZA SUHEL INAMDAR: Hi everyone, my name is Aleeza and today I want to present my masters thesis which talks about operator perspective on BGP security solutions.
So, to begin with a little bit of motivation as to why we chose this topic. This image that we see on the left, it's cute, it's colourful, it looks simple, and this is a picture from our data network lecture that I had at my university, and when I was going to the course I thought okay, how hard can this be? And a couple of month down the line, we had another course which was a little more hands‑on, we had our AS connected it to the Internet and did a bunch of stuff, and the picture on the right was the end result, and then I thought okay, not as easy as data networks said it was going to be.
And that's when we realised that there is this potential gap between academia and the industry which then raises the question whether academic solutions are practical or not.
There are ‑‑ there are a number of works out there on BGP security, but very view of these are actually put into practice, and we want to understand why.
Operators might be sceptical.
So coming to operational realities in BGP, finding good people is hard because younger professionals are ‑‑ want to work in more sensational fields like AI or Cloud computing because apparently BGP is uncool, and again, cost cutting is another issue that we see in organisations these days where senior professionals are let go because organisations want to save on their revenue.
And lastly, BGP is hard. That's why operators tend to stay far away from it. So far that sometimes they want to outsource it to third party vendors for convenience.
So then, we decided to collect a bunch of operators and discuss this with them. We conducted focus groups and spoke to these operators about this potential gap between academia and the industry. We asked them for their opinions on academic papers, and this is what they had to say.
So, academic papers lack practicality simply because sometimes researchers make these simplified assumptions while conducting their research, they don't account for these variables that, hidden variables rather, that are present in reality secondly, academic research doesn't account for organisational restrictions and other overheads. So then while trying to implement a solution in practice, there might be issues with respect to scaling and deployment.
Additionally, there exists an access barrier to research papers because sometimes you need paid subscription to say access these because they are not open access.
And the format of these research papers. So, it was told to us that these might not have the best format. All the practically relevant information is embedded within, or around other text.
And lastly, academic papers are sometimes just written for the sake of it. So a very good example of it just might be Ph.D. student who is trying to work on that very last paper to finally get their Ph.D. and then they graduate, they move on, what happens to the research? Another example of this is in academia, academics don't say put, they change their affiliation and then this also hurts the research.
So then, our participants seem to believe that academia has Ivory Tower syndrome. There is a misalignment between both the sectors ‑‑ there is a misalignment between both the sectors, but why?
.
Firstly, as we discussed the barriers to practical implementation. Accessibility. This was a very interesting find because one of our participants said that the real problem is not access. ‑‑ sorry the real problem is not interest, rather access to these papers.
And lastly, funding. So it's really important to ensure that funding does not come from for profit organisations because then this might lead to results being skew.
This brings us to a situation where we have researchers and then we have operators. Researchers are willing to invest a lot of time, a lot of effort. They want to understand a problem and provide a meaningful solution. However, they lack practical experience and also guidance from people that actually run networks, and then we have operators, they want to keep their networks up and running and to do that they want easy and quick to deploy solutions.
So then what can be done to bridge this gap between the two sectors?
.
Our participants proposed some really nice recommendations. Firstly to propose modular and incremental solutions in such a way that these could be easily incorporated within an organisation, but again the way academia is set up, this might not be that easy because then you don't have this one big impactful solution.
Next is improved communication and collaboration, for example the RIPE meetings. These are a really good platform for people from both the certificate, to come together, collaborate and have meaningful discussions.
Another recommendation is to enhance the way you present your findings. So, maybe a white paper or an article as published on RIPE Labs where you say hey, this is my solution and so are the steps to deploy this. Simple.
Next, again as I mentioned earlier, funding. So, ensure transparency while gaining funding for your search.
And lastly, and also an important point, is to integrate more hands‑on courses within an academic curriculum.
But again the way with academia works, it might not be as easy to set up these, or to work on these recommendations. It would require a lot of dedication and a long term commitment to actually make meaningful contributions together.
Another thing I would like to say in conclusion is, me being a student, I now see a really big gap between the skills that students possess and the expectations of the industry, even when they want to hire junior engineers, it's it's a very visible gap which is why it's really important to start integrating more hands‑on courses in your academic curriculum. With this I'd like to conclude my presentation and I am now open to any questions.
BRIAN NISBET: Thank you very much.
AUDIENCE SPEAKER: I'd like your talk here, and I pick up you are saying that you are also saying that you are learning more about the practical side and you see what's different. When we look at what we can do to get more young people in networking, do you worry there is a risk that someone with a masters degree or Ph.D. would want to do academic research but wouldn't want to come in and do plugging and configuring?
ALEEZA SUHEL INAMDAR: Yes and no. Because if ‑‑ even if the someone without a masters degree or a Ph.D., there are a lot of people, I have seen, that are, they like poking around and they do things in their free time, and that's something that is, it's worth something.
AUDIENCE SPEAKER: Do you think we could ‑‑ anything we could do to get people both in academics and pre‑academics more involved in the industry?
ALEEZA SUHEL INAMDAR: One thing that could be done is have people from organisations collaborate with universities, like come and give talks and also hire, right, like when coming and talking about somebody, maybe there is a requirement in the organisation, you give a chance to people that want it.
AUDIENCE SPEAKER: Lisa. I have more like a comment actually. But because you just said we need it more practical courses and like academic programmes. I just finished my masters last year and they are actually in the process of removing more hands‑on courses for now because it doesn't fit the master university model. So it's more of maybe a depressing comment but, so, yeah, sad to see but I am also doing BGP security search right now so I see these problems as well. Thanks.
AUDIENCE SPEAKER: Hello. We see the gap between the academics and the industry and we try to do something, we introduce the research fellowship, I don't know if you are aware of that, but my question here: What more can organisations do for that because they sit somewhere in the middle?
ALEEZA SUHEL INAMDAR: Like I said, you get involved with universities. You do internships, you try and hire students that fit in whatever domain that company works for. You do talks at universities. This would really help. It would help me personally.
AUDIENCE SPEAKER: Darren Clarke. We currently work with a local university that's just down the road from us at our office in Cambridge, and doing the things that you have actually suggested there, we have hired the last three NOG engineers directly from university and we also do internships as well so we actually train potential engineers on what the industry really looks like, and I completely agree with you, they are very surprised at what they see when they come into the NOG, compared to the things that they are actually being taught. So there is definitely a gap there and something that does need to be addressed.
Just one very quick point, again on your earlier talk, was that one of the problems that in general I saw when I was at university, is any research that leads to any product you develop in university is then covered by the university's intellectual property, which means it's very hard to get it out into the real world. It's maybe something to think about.
ALEEZA SUHEL INAMDAR: Yeah, but when you are hiring somebody who is developed something like that, it's going to be beneficial to you as well in the future.
BRIAN NISBET: And I think a lot of this also, it would be difficult to not draw parallels with Maria's talk from earlier with everything we're talking about, it's learning these practical skills that exactly the kind of thing we need for, you know, the freedom Internets of the future, all of these other things, so I think it's very important.
So thank you very much.
(Applause)
.
And for our last lightning talk, we have someone who has been on this stage once or twice before. We have Andrei talking about MANRS evolving.
ANDREI ROBACHEVSKY: Hi. I'll be giving you a very short update what's happening in the MANRS land.
Okay. So, what's MANRS? MANRS stands for mutually agreed norms for routing security and it's industry effort that is running for more than a decade to improve routing security in the Internet. And it is based on two pillars basically, it's based on what we call norms, undisputed baseline and this is not best current practice, that's minimum baseline, that is defined by MANRS actions, and the growing community of participants that stick to this actions, those requirements, and demonstrate their commitment and the, this commitment is verified and displayed on a MANRS website and MANRS observatory.
And in those updates, we usually talk about the if I remember part, about or the second part about the community, what's happening, how it grows, and it does but today I would like to talk about the first part, about the norms, and those norms are defined in foundational documents, what we call MANRS actions, and for those who follow MANRS, know that they are four programmes in MANRS, for network operators, for IXPs, CDN and Cloud providers and equipment vendors. The requirements are defined in those documents. There is an important document called MANRS chatter which is a governance document for the initiative.
There are also a couple of guidance documents developed by the community such as network operator guide, IXP guide and also some recommendations developed by the CDN and Cloud Working Group.
So, all those documents in fact you might ask how they came up. I mean who compiled those documents and imposed on the industry? So, they were actually developed in a very open, collaborative way. What groups were created, task forces were created, they got sufficient review, there were last calls, but if you look at the website, you don't see this. I mean, really, this process is not well documented, theres no evidence that it was indeed developed in this open way, so you may warned what consensus it is. Is it just a group of ten people, 100 people, how wide of the review? So that's one of the problems.
Another problem is MANRS is maturing around MANRS is a well‑known effort and is being referenced in many documents of policy documents, recommendations, I think Valerie mentioned this report on resilience, even there MANRS is mentioned. But how it is referred to? Well it is referred to the website maybe as an effort. But not as a set of requirements you can practically use for instance your procurement practices.
So, taking this all into account, we came to the conclusion in the community that we need a well documented process that we need also a document series that enables stable researches to the MANRS actions, the MANRS requirements.
So, that's what we call a MANRS development process. It's sort of a standard development process, but in the MANRS land, and we did did this not to improve existing standard or policy development processes. Actually we picked up the pieces from existing stuff, from RIPE PDP, from the IETF, standard development process from the first standard development process, and compiled something that we own and control. That is really not to compete with any other standards but something that we can track and demonstrate how those documents are developed.
So, in a nutshell, very simple, and you can recognise many of the processes follow the same thing.
Proposal, announcement, development, public comment, approval, publication.
Now, this is a more detailed diagram, so they are writing specifics of the MANRS. One of the specifics for instance is that there is a MANRS community which is a bigger community of network operators that are interested in routing security and MANRS. And a smaller group that are MANRS participants, they also participate in improving MANRS standards. That's because we would like to get more expertise from the outside but because the requirements, normative documents, that are developing MANRS are imposed on MANRS participants, we need explicit approval from that group of people. That's just one of the details.
So, the process is nothing as a process. The process opens a way to review in a very transparent way of how MANRS actions. And we had the situation when people come and say, you know, we would like to suggest some improvements to existing requirements, so maybe improve some of the requirements. Some of them have been like six, seven years old. But what is the process? I mean, do we just write to Andre and he makes a change in this document? No, that doesn't work this way. So, really, this process opens for us and opportunity to go and review all those documents. We actually killing two birds with one stone. Because we are reviewing this document improving the document but we are also adding the document to the document series. So from now on people outside or efforts outside can reference these documents as a stable MANRS standard.
So, there is some suggestions. We are just starting discussing the stuff. MANRS actions, I think for many of them it's time to update. For instance, for network operators action, I think it's time to mention ROV as one the entry requirements. Not all of MANRS participants are implementing ROV, how you grand parent this stuff, all this needs to be discussed.
RPKI, well, there is a little secret, MANRS is raised from promoting RPKI but RPKI is not mandatory in MANRS. Oops. We need to change that of course.
IXP action. I know that some people in IXP community are thinking about introducing some or more stringent requirements, how to use IRRs in full back from the RPKI. So that could be incorporated there.
I presented some time ago an effort what we called MANRS plus and now this document has been published and it's well thought through by the community document of some additional requirements that could be integrated in MANRS actions. So there are some ideas that we can use to, and use the MDP to actually put this in practice.
Now I mentioned the MDP that there is a community and participants. I have to say that originally, MANRS community and MANRS participants were the same and they were used interchangably but we're changing this, we are changing this with the MDP because we would like to get a broader feedback from the community, broader expertise, and also, you know, it should be a result not of the consensus of a small closed group like MANRS participants, but more reflecting global consensus of the community. It should also be an aspiration from people that are wanting to join MANRS but not in MANRS yet. So, MANRS community is actually broader. It's wider. Anyone who is interested in routing security, anyone who wants to put their efforts into promoting MANRS are welcome to the MANRS community.
And I think that's it. Stay tuned. And if you would like to join MANRS community, go to the website and join it there without becoming a MANRS participant necessarily.
(Applause).
BRIAN NISBETT: Thank you very much. Do we have any questions? I am going to ask a question.
When is MANRS moving to a social media platform that isn't X?
ANDREI ROBACHEVSKY: Well...
BRIAN NISBET: I should ask that of more people. Anyway it's cool, I am not going to put you on the spot.
ANDREI ROBACHEVSKY: You did.
BRIAN NISBET: I did, apologies, but I didn't ask you to answer. In all seriousness, I am serious ‑‑ no, any other questions, folks, that are actually about MANRS? Again, so, contacting you and I presume at future RIPE meetings and other things there will be opportunities and you will be around all week to talk about this, and Blake is standing up.
AUDIENCE SPEAKER: Blake Willis. I see a lot of people walking around with MANRS T‑shirts, is there somewhere I could get one of those? They are awesome.
ANDREI ROBACHEVSKY: That's another difficult question because my answer is no. You have to grab someone you see, negotiate with them. We didn't bring T‑shirts this time.
BRIAN NISBET: This is not a suggestion you should be ripping T‑shirts off other RIPE attendees. There is a Code of Conduct.
Okay, in which case, thank you very much Andre.
(Applause)
.
So, just to say that that concludes our session for this afternoon. To remind people that the PC nominations are open until tomorrow afternoon for nominations. If you are nominating someone, please make sure they know you are nominating them and remember there are two BoFs this evening, there is the BCOP Task Force BoF and a BoF on ICP 2, they are at 1800 in the 2 rooms. So thank you all very much.
.
(Coffee break)
