ARIN 51 Public Policy and Members Meeting, Day 2 Transcript - Tuesday, 18 April 2023 [Archived]

Opening Announcements

Hollis Kara: Welcome to Day 2 of ARIN 51.

(Applause.)

Again, thank you to our elected volunteers. I hope everybody got a chance to chat them up at the social last night.

Now, again, we’re going to fly through the reminders just to make sure we’ve got a level set for today. Chat is for chat. Q&A and raising your hands are for entering comments or questions that you’d like read into any open discussion period or would like to contribute.

Again, the Virtual Help Desk is available until 9:30 this morning and will be available on breaks and during lunch if you need assistance there.

In person, again, you are welcome to join the Zoom. Please just make sure you are completely disconnected from audio if you choose to do so. But you are welcome to hop in there and talk to folks in chat if you so desire.

The navigation is on the website. Lots of options. You can find pretty much anything you need to know; and if you can’t, let us know and we’ll make sure we add yet another thing to the navigation.

Reminders: Producers and hosts will coordinate discussions. We’ll be queueing for the mics and for remote participants. When you do enter a statement into the record, please make sure to start with your name and affiliation. That counts even if you’re typing. And we’ll tell you when we’re ready for you.

We are recording and live streaming. The slides are available through the meeting materials page. So you can view PDFs there if you would rather view them locally rather than looking at the screens up here. And the live transcript is running in the link — that’s also available on the meeting materials page.

We will be having virtual table topics today during lunch. So a few of our lovely AC folks will be stepping away and not joining us on the promenade, but rather hosting virtual chat rooms to cover some of the topics that we had at our in-person lunch yesterday. So it will be a great opportunity for our virtual participants to have a little bit more informal discussion on a range of topics. So we’re looking forward to that. And, please, hope that you will join us.

Again, Jesse has still got a big silly smile plastered to his face. He’s out in the foyer running our user testing table. We also have an option to participate with user testing virtually. You can go to ARIN.net/testdrive. There are a few online testing options that you can access there. There’s also a QR Code if you prefer to subtly go and grab that out at the desk and just something to do to kill time at the airport on Thursday on your way home, or Wednesday, whenever you’re departing.

Again, a big thank you to our sponsors. Charter Communications, for our network. IPv4.Global by Hilco Streambank — I don’t know why that’s so hard to say — our bronze sponsor. And Google, for our webcast. If I can get a round of applause for our sponsors.

(Applause.)

A quick tour of the agenda. This morning, we will start off with a riveting Routing Security Update from Brad Gorman. Followed by updates from our 2022 grant recipients, the current… no, it’s 2023. I don’t even know when I am anymore. Our grant recipients currently in progress. We’ll get updates from all three of those organizations.

And then after our morning break, we will have a bunch of interesting presentations followed by our IPv6 Success Stories keynote. We’ll be celebrating 25 years of production of IPv6 at Virginia Tech with a bunch of the folks that have been making that happen.

We’ll break for lunch. And when we come back, we will have an IPv6 Success Stories panel. We have five folks from the community who have successfully implemented IPv6 at a range of organizations. And we’ll be hearing from them what they’ve learned along the way.

Then we’ll take another break and wrap up the day with updates from our fellow RIRs, from IANA. We’ll do a Regional Policy Update, and then close the day with an Open Microphone.

With that, we’ll get started. Brad, if you would like to join me on stage.

You’re right, we really do need play on music. We’ll work on that for next time.

Routing Security Updates

Brad Gorman: Thank you, Hollis. Again, my name is Brad. And I am a Product Owner for the routing security services, it’s in the title. But just to give you an explanation a little bit of what I do.

I kind of have two functions that I serve here in this role. I come to community meetings like ours or to NANOG and periodically other IETF meetings and RIR meetings where I come and listen to suggestions and requests and feature desires and maybe look at some of the fixes that maybe we can offer to our users of our services.

And then on the other side, I bring those kinds of requests back in and manage the priority and processes that are associated with getting those things developed for you.

Let me just go ahead and get started. This is the update that we’re giving on what ARIN is offering you today. I’ll give you a quick update on the team, go a little bit more in depth into how you and the community are using the RPKI services here at ARIN, and then go over a few of the recent delivered features that we’ve given everybody, and then give a little hint of what’s coming.

I’m going to give Jesse one more plug. He’s had some of the actual screens and more of the pretty things to look at that go along with the words that you’ll see a little bit later.

Team update. New to the team is Nathan Newman, who couldn’t travel with us today. But Nathan has been at ARIN for a couple of years. Probably approaching three.

His primary role has been responding and answering technical questions that have come into the ARIN Registration Services team, through the chat, through Ask ARIN questions or other methods that come in.

If you had the opportunity or questions that have come in regarding RPKI or the RIR, or DNSSEC, you’ve probably already talked with Nathan. Really excited to have him on the team. And as our team mail alias, feel free to ask a question, whether it’s off-the-cuff or something specific, we’re here to help. That’s one quick way to get a hold of us.

Where do we look in the ARIN region with RPKI adoption? For those who may not know what RPKI is and what it means, it is the Resource Public Key Infrastructure, and what it means to you and to the Internet, it’s a method of using cryptography to sign Internet number resources that have been allocated under, in this case not just by ARIN, or any of the RIRs and it allows them to make an authoritative statement about those resources on where they should come from, where those prefixes are going to be originated.

And the enhancement and the capability of RPKI allows people to make those statements with the assurance that those resources are coming from them. Those are their resources and they’re the rightful holder of those resources.

General benefits of RPKI: Give operators another tool in their routing security bag that will allow them to make better informed decisions on how to handle and how to accept or process routing updates that they hear on the Internet.

The key component of that is what’s called the RPKI Validity Statement. I’ll go a little further — explain what the validity statement means. But it comes to using RPKI information that users have created and matching it up with what goes on in the Internet table and be able to make better decisions instead of just what they generally hear.

Another benefit is it allows resource holders to make these statements and to kind of protect those resources from being impacted by either human error, mistakes, with announcements going to the Internet, and be helpful in blocking the nefarious activity of people, the bad actors out on the Internet, from trying to disrupt their service or otherwise reachability to your resources.

And thereby it reduces the attack vector or attack surface afforded to those people by you, the resource holders, making statements about what you have.

There are three key components. But many more pieces to it. I just wanted to give a quick explanation of what these pieces are and how they might work together, how they do.

The first is a Trust Anchor, and the RPKI uses exactly the same infrastructure and same cryptography method that HTTPS uses, the same X.509 protocol. And as a Trust Anchor, ARIN is the one who authoritatively responds to any sort of requests to confirm these resources came from this person and they’re valid and you should go ahead, and we confirmed that that request is correct.

Then there’s the Certificate Authority which handles the cryptographic components of the RPKI, as I started to explain. And then the absolute key piece of RPKI is these Route Origin Authorizations, or ROAs. This is what you, the resource holder, go and create. That’s where you make that statement about here are my resources; they should be coming from this location, and I am the rightful holder of these.

And that is a broad use across the Internet. Everybody may not have foreknowledge of your resources and how you want to use them. And these ROAs are one way of allowing you to do that.

There are two basic services of RPKI. There’s Hosted and Delegated RPKI. I want to go into those two and the service that we offer here at ARIN. Hosted is a method that is recommended for people starting new.

It’s a service where ARIN handles most of the pieces of the RPKI pie. We are the Trust Anchor. We were always the Trust Anchor for resources that are ARIN resources. We’re the Certificate Authority. We handle all the cryptographic component, making sure that all the pieces line up and that the other pieces of the objects inside the RPKI are justifiably and appropriately created. And then there’s the repository.

Any of the objects that you create or other pieces of the RPKI, we maintain those, hold them, and then announce them to the outside world so people can make use of them.

The use case for this, again, it’s the simplest form of getting on your feet and starting to use RPKI. As the resource holder, your responsibility is creating these ROAs, making these statements about your prefixes and where they should come from, that’s up to you. But that’s all have you to do, which is great.

And as of the beginning of this month, more than 98 percent of ARIN organizations that do use RPKI services that we offer use Hosted RPKI.

The second form or second offering for RPKI is called Delegated RPKI. You may have heard up/down. It’s just another way of describing how this service works.

When an Org decides to deploy a delegated service, a delegated instance, they have taken over all of the responsibility for their cryptography. They run the Certificate Authority.

They hold all of the pieces on how to create their ROAs, the lifespans that they associate with them. Essentially, they have everything in their hand capable. And it’s truly up to you to create, maintain, monitor, hold your repository, report the information that’s in there. And very small use case, both here at ARIN and across the Internet, use a true delegated instance because of the heightened requirements of running all those pieces and all those components.

There is the Trust Anchor function by ARIN. We still have to authoritatively respond on users who have these resources and say, yes, these are genuine ARIN resources, and, yes, the right people are making these statements. But beyond that, it’s all on the organization running the delegated instance.

There’s a third piece or component or a second piece of running a delegated service. You may have heard the term “hybrid RPKI.” And really the way that works, and what it means, is an organization still runs their delegated certificate for you. They’re still responsible for the cryptographic piece of it.

They still have the ability to create and maintain their ROAs, but they have opted to use a third-party product offered by ARIN called a Repository and Publication Service, or RPS. And we run their repository. We make sure that those high-availability pieces of the RPKI are there, and were maintained and rest assured that it will be there all the time.

That is kind of the third version of how RPKI works. We enabled that service at the very beginning of 2022. And there has been uptake as organizations decide that, one, they want to do delegated, and two, use our service for the repository.

Let me get into the numbers, a little bit of the nitty-gritty on how we as a community are using and adopting RPKI. With the hosted repository and the RPS repository, that hybrid repository, we maintain two separate entities.

And the primary or hosted service in that repository, at the beginning of this month we had almost 55,000 organizations who had signed up to use RPKI. I’m sorry, we have 55,000 ROAs in our repository. Organizations who have created these objects have been put in, and we are appropriately responding to any requests for that information and people have made that decision to cover their resources using a ROA.

Within those ROAs, if there’s the prefix or multiple prefixes that are in that ROA, those are converted into what are called validated ROA payloads. All that cryptographic component has been checked and approved, and those ultimate pieces of data are what operators use to make that RPKI validity discovery and use them in their routing security.

Just as a comparison in the numbers, the RPS repository, significantly smaller, but it’s actually shown a really good growth pickup since we turned it on. And the number of objects that are in there by percentage are growing much faster than in the hosted repository.

Valid route announcements. I spoke a little bit and brought up route validity. Route validity can be measured in three ways: Valid, invalid or not found. Two of them are pretty straight forward. Valid and invalid.

Not found is a circumstance where there’s a prefix that’s visible on the Internet Routing Table but there’s no corresponding data inside the RPKI infrastructure. There can be no determination of what it is. That’s effectively how the Internet works today. Your BGP routes are announced to the Internet. It’s the way it’s been configured and run for years, that’s how your route is treated.

But as of now, it being the end of the month, almost 29 percent of routes on the Internet, the global table, that have ARIN resources in them, are marked as valid. It’s not really great. We really need to do our part and go beyond that.

But that does represent a percentage increase of six and a half percent for v4 resources that are covered in RPKI. So we’re making progress. But as a number, 29 is one that we’d like to grow from.

On the flipside, v6 resources, of those announced to the Internet, we’re approaching 50 percent, which is great. Glad to see it. Glad to continue to see it come up year over year. But we all know, as operators and people who have run networks, 75, 80 percent of the Internet is v4 addresses. So let’s make a push for that.

And those graphs down at the bottom, the text is very small, they came from a NIST RPKI tool. You can go to that link. It’s a great resource for anybody who wants to get a view of either RIR specific or global numbers with utilization.

A lot has been said about — ARIN is for some reason holding back deployment of RPKI. And a lot of that does have to do with our resources need to be under agreement to use routing security features, whether it’s RPKI or authenticated IRR or DNSSEC services, which actually recently changed.

But when you go and into consideration of eligible ARIN resources that are being announced out to the Internet and being marked as valid, 45 percent of those resources are marked in green. So it’s a much better representation of, as we are producing and using these services, our commitment as a community to using this tool and moving forward.

There are a couple of little asterisks down there I want to explain. Eligible, is they’re under an agreement, your resources are under an agreement. And then marked as valid, of course, you have to have an exact match between what you’re announcing in your routing table, your routing BGP, and then there is a corresponding match in a component, in a ROA, in a VRP, inside the RPKI infrastructure. That’s how you get to the definition of valid.

Here we go. The number of organizations that have signed up today to use RPKI inside of ARIN is roughly 3400. Number is definitely going up. But as you can see, the percentage between hosted and delegated, there’s that 98ish percent of people who have signed up for one service or the other. And then the repository service. So those delegated customers who have chosen to use our repository versus running theirs, that number is now 31.

And just as a reference, the graph on the righthand side of the screen is all of the other IRRs, excuse me, RIRs. ARIN is the blue line at the bottom or close to the bottom.

What I wanted to put out is it’s general, up and to the right, growth of deployment and uptake of RPKI globally.

It’s continuing to happen. For those of us in the community, that is absolutely a great thing that we want to see, and we want to see more growth.

We’ve released some new features this year. And we have a process and things in the flow that we’re going to continue to give you some of these new capabilities and features that will make it easier for you as resource holders in ARIN and users of our tools, the ability to further assist their deployment.

Some of them are usability. In February, we gave people the opportunity to go and create their ROAs, their objects in the database. Used to take five clicks to get there. And I was frustrated. I know you were frustrated. I heard it many times. And starting in February, we changed navigation to go and allow one click down to go into RPKI and then another selection to go in and say here are my RPKI resources, let me get started. Now it’s two clicks away.

Another feature that we added was making it easier to switch between multiple organizations. If you’re linked to more than one, it used to be go back around in that circle to go get there. In the presentation that we’ve got now and where you go to manage your devices, we have a little dropdown, internally we call it the Org picker.

If you selected one Org to do some changes and you want to switch to another one and make resource changes all you do is you click the dropdown and go from one organization to another.

We’ve got a lot in development. I’ll give another plug for Jesse at the user table outside. He has a picture, explanations and examples of where we’re going with some of this development.

One of the big things that came out of the announcement earlier this week, late last week, ARIN is no longer requiring you to sign ROA creations. There’s no more public/private key. That has been a number one, kind of, difficult thing for people to get over. And it’s been something that we’ve listened to for a long time. And we finally took that requirement out. These are features that are coming May time frame and later on this year.

We are now going to be auto renewing our ROAs. So historically, when you create ROAs, whether it’s through the user interface or using the API, there were fields where you would set beginning and end validity dates of those ROAs, essentially creating it, setting the lifetime of your ROAs.

Well, that has been either a management nightmare for Orgs with lots of ROAs that they’ve created. Or maybe for organizations that aren’t as well versed or understanding how these ROAs apply to them and their resources and how things are improved on the Internet by using them, if they expired, there could be catastrophic results.

We are now on a 90-day cycle reupping a ROA and thereby they become evergreen. They’re not going anywhere. Still, it does not stop you from manually going in and deleting your ROAs if you so choose. You can still take advantage and make use and give that lifespan definition, by going in manually and deleting them.

Along with these features, we’re creating a new API, new RESTful interface, for creating and managing these ROA components.

There’s development that’s coming in the community and the IETF community for new use cases for the RPKI structure. And this API, as it’s been written, is futureproofing, giving the capabilities to take advantage of this new development that’s coming in the community and further being developed.

This, I believe, is where I’m ending up. I’ve never been accused of talking too slowly. I welcome to answer your questions and please come up to the mic if you have questions.

Hollis Kara: Mics are open if you have questions for Brad. We do have a question that came in over virtual, if we could start there.

Beverly Hicks: Thank you. Chris Hansell from Rackspace U.S. Does ARIN charge a fee for any of these services related to ROA management; and, if so, can you explain?

Brad Gorman: Fees, no. The services are available to any organization in good standing who has signed and has their resources under one of our agreements, RSA or LRSA agreements. But once those criteria have been matched, you do not pay additionally for RPKI or our authenticated IRR services.

Beverly Hicks: Thanks.

Kevin Blumberg: Kevin Blumberg, The Wire. Thank you. Thank you, thank you, thank you. Can’t reiterate it enough.

The whole private key and dealing with that was an absolute barrier to nontechnical organizations who now had to wrap their head around RPKI. The fact that you have had ROA-thons to help people set up, where it could have been a button click, is a great example of that.

So this will hopefully save yourselves, your help desk a huge amount of time. But more importantly, this is a massive improvement for the deployment of RPKI.

I did notice you’ve got a 90/30 sort of on the renewals, 90-days certificate, 30-day renewal. Very, to quote, let’s encrypt sort of timing, fine. But what is the notifications around that? And as in, will you be sending, if you’ve renewed your ROA, will you be sending your ROA is now in some weird state and has 20 days left on it, open a ticket, there’s a problem?

What sort of notification system, because out of sight, out of mind is great until something stops working for some reason.

Brad Gorman: Excellent question. So the two different ways that users will be notified, in a sense, that their ROAs either have an end validity date assigned to them, whether its ROAs created well into the past — I’m going to give a little bit more because there’s something behind that.

If they go into the table inside of our hosted product or use the API, they can now determine and now see that your ROA is being autorenewed. That’s the first piece of, hey, you know you’re in part of the process of the service that ROA will not end.

At this point, today, users are notified beginning at 30 days and up towards 10 days, and then more regular kind of pokes going, your ROA will end on — your ROA has 10 days. Your ROA has five days.

And one of the things that we noticed, when we were working on sunsetting our nonauthenticated IRR, we sent notifications out to all points of contact linked to organizations that had the authority to make changes there, saying, hey, this is coming and we made those announcements starting nine months ahead of time.

We know for certain that some organizations haven’t updated their contacts, or the contacts that are there are no longer with the organization.

Auto reroll is part of the safety net that we’re doing for organizations that maybe don’t have those kinds of contacts. The attempts to reach them that maybe are falling on deaf ears or email boxes that no longer exist, that’s one of the reasons we’re doing the reroll.

Kevin Blumberg: On an auto reroll, you’ll be looking at why something isn’t renewing for some weird odd reason?

Brad Gorman: Right. If it goes through and it passes that 30 days, we’ll certainly know if there’s something going on in the repository that’s otherwise not serving that auto reroll function. But those subsequent email notifications, we have taken those out. And arguably they will be triggered well before — we will fix the auto reroll function well before they take it down to that.

Kevin Blumberg: Thank you.

Brad Gorman: Question in the middle of the room.

Louie Lee: Hey, Brad. Thanks for the presentation. Can we go back to slide 25 real quick?

Brad Gorman: 25? I see.

Hollis Kara: Pretend we don’t know you. Can I get your name and affiliation?

Louie Lee: Oh, thank you Hollis.

Hollis Kara: Thank you.

Louie Lee: Louie Lee, Google Fiber. Also Louie’s hat. That graph, what’s causing the dips during the months?

Brad Gorman: So a very large customer of ARIN with a great deal of number resources had a process, an automated process, in place where they would create, give a very short lifespan to their ROAs and allow them to expire. It was on a 15-day cycle over a 30-day period where they would create and ROAs would get deleted.

Because of the volume of ROAs they had in the system, it literally showed up and affected what this graph looks like. Now, near the right-hand side of this graph, subsequent spike, subsequent drop, and then back to what would appear to be a normal glide path. That particular organization, list of organizations has now deployed, taken away from our hosted repository and deployed, their own delegated instances.

So they’re no longer — their process is no longer affecting what the graph looks like.

Louie Lee: Have you talked to them to see why they would want to be withdrawing like that?

Brad Gorman: It was before my time, but what was going on is, in certain components in the RPKI, there is a zero list, which is kind of the ledger of ROA creates and ROA deletes.

And in that ledger at ARIN, there’s 20,000 lines in that ledger. Since they had such a large number of ROAs, and when they would go create, and if they had customers that came in or withdrew and those resources that had been assigned and then subsequently taken away caused a great deal of churn in their numbers. I don’t believe they ever reached that 20,000 full ledger, but it was their way of maintaining the lifespan of their ROAs so that they would not be impacted by that number.

And just to rest assured, this is by far the only organization that we would have considered approaching that limit. Most organizations have maybe a few hundred of things in that also.

It was kind of a joint interface between ARIN and the organization to do it. And this is what they came up with on how they would manage it. That was their choice to do it this way.

Louie Lee: Sure. Thank you.

Hollis Kara: Do we have any other questions?

Nope. I think we’re done. Thank you, Brad.

(Applause.)

Going to click. And next up we’re going to start off with our Grant Reports. I’d like to welcome Carolina Caeiro who will be joining us virtually and Mark McFadden who’s making his way up to the stage to give an update on their grant report on routing security in the ARIN region.

Community Grant Reports

Routing Security in the ARIN Region

Carolina Caeiro: Good morning, everyone, from sunny Barcelona, here in Spain. My name is Carolina Caeiro. I’ll be presenting today with my colleague Mark McFadden, our research and data analysis on RPKI adoption and routing security in the ARIN region. And I see Mark is on the stage, if you want to say hi before I dive into the presentation.

Mark McFadden: No, no, just go ahead.

Carolina Caeiro: Wonderful. All right. So next slide, please. All right so to get started, just a quick word about the project we have been working on and under which we’ve conducted the research we are presenting to you today.

So our work on RPKI adoption is part of an initiative we are conducting with the support of ARIN’s Community Grants Program. The goal of our one-year project with ARIN is to showcase data on RPKI adoption and routing incidents in the ARIN region. And by sort of collecting, analyzing, and providing open access to that data, we’re hoping to shed light on the current status of RPKI adoption in the ARIN region and hopefully encourage greater academic and industry engagement and discussion on current routing security practices.

We are not the first, of course, to gather and analyze data on routing security and RPKI, so specifically we are hoping to bring value added with our initiative is in providing geographic analysis on RPKI adoption for the ARIN region. That means looking at RPKI adoption for all the countries serviced by ARIN.

Our project also entails ingesting and conducting live analysis on RPKI adoption. So we’ll be launching a live report showcasing the various indicators we’re looking at as part of the project, with focus on the ARIN region of course. That will be available on a regular basis.

Lastly, anyone interested in sort of playing directly with the data that we have gathered on RPKI adoption, you can sign up for an account on our data analytics platform or DAP.live and do your own analysis there.

I will be showing in just a bit what our platform looks like and the type of queries that you can do on there.

Next slide, please. So a quick word as well about who we are, the organization behind this project. So the project is spearheaded by the DNS Research Federation. We’re a not-for-profit organization based in the UK. And our goal is to contribute to making sense of the Internet, really. So helping people understand the role and impact of the Internet’s unique identifier systems on today’s most pressing cybersecurity and cyber policy issues, including functions such as routing security.

We have three areas of activity that you can see on the screen, and these projects sort of straddles across the first two. So facilitating access to data, in this case on RPKI adoption. And the second facilitating the understanding and analysis of that data through outreach and research.

Next slide, please. So about today’s presentation, what we’re bringing to you is some of our preliminary analysis based on the data that we have collected and processed on RPKI adoption in the ARIN region.

Here I will mention we have contrasted some of our findings with other initiatives on RPKI adoption such as the MANRS initiative and the NIST of which Brad was just showing some statistics just now. We are obtaining very similar results and we’re still finessing our algorithms, sowe’ll be showing you, again, a bit of what our data tells us.

So we’ll start off by walking you through our global steps on RPKI adoption and validation results and how the ARIN region is performing as a whole.

We will then take a deep dive and look at the dynamics within the ARIN region and introduce some subregional trends that we’ve, yeah, seen emerge in the data. We’ve also taken a close look at invalids within the ARIN region, sowe’ll be presenting and discussing patterns we’re observing when honing in on invalid routes in the ARIN region.

And then we’ll wrap up with a few words on the methodology that we’ve used and pose really a question to the audience about whether we should be considering additional or complementary ways of measuring routing security that take into account the level of protection of end notes as the result of RPKI adoption.

Next slide, please. So let’s dive into some of our results. What you’ll be seeing in the next couple of slides are screen shots of our data analytics platform, DAP.live, which is a tool we’ve used to conduct our analysis of the data and a tool that you can get access to do some of, some data processing yourselves.

So while my colleague Mark will speak about our methodology and data sources in greater detail, just to give you a sense of what we’re seeing here, what we do on the platform is ingest raw BGP data and then analyze the RPKI validation results of observed BGP announcements on the Internet.

So the first question we set out to explore with our research was, what is the level of RPKI adoption in the ARIN region? And I think Brad was already hinting at some of the results we’ll be presenting in the next couple of slides.

So in order to answer that question, we started by looking at what was happening at the global level beyond just the ARIN region. So what you see here is the first indicator we set out to consider, which is RPKI coverage.

RPKI coverage considers for each unique prefix, origin Autonomous System pair observed through our collectors, whether the prefix in question is RPKI protected.

So as Brad was explaining, we consider a prefix is protected when covered by a route origin authorization, or ROA, which is a certificate that confirms whether an Autonomous System is authorized to originate a particular IP address prefix or set of prefixes.

So our global coverage analyses indicates that for IPv4, 43.4 of (indiscernible) prefix origin pairs are RPKI protected, so slightly less than half, whereas for IPv6 we observed nearly at 50 percent RPKI protection at the global level.

Next slide, please. So we then have a look at ARIN and we found that for IPv4, RPKI coverage stands at 25.6 percent. So nearly three-quarters of prefix origin pairs observed are actually unprotected in the ARIN region.

This is, we can say, well below the global RPKI coverage level of 43.4 percent that we saw in the previous slide.

For IPv6, ARIN values are on par with global coverage levels. We found that 50.5 percent of prefix origin pairs observed are RPKI protected, so that is pretty much in line with the global coverage level of 49.9 percent shown in the previous slide.

Next slide, please. So the second indicator we looked at was validation results, which beyond the level of RPKI coverage tells us what percentage of the observed prefix origin pairs are invalid.

So in terms of our global validation results, we found that for IPv4, 41.7 percent of the observed prefix origin pairs were valid. Therefore the origin Autonomous System was authorized to announce a specific set of prefixes. 1.6 percent were invalid, meaning the announcing Autonomous System is not authorized to announce a specific prefix or set of prefixes.

And therefore that means that we could be dealing with either a misconfiguration or a hijack. And then 56.5 of the observed prefix origin pairs are not found, meaning that no information has been found to validate whether the Autonomous System in question is allowed to announce a specific set of resources, and therefore we can say that those resources are unprotected or vulnerable to hijacks.

For IPv6, we see that of the nearly 50 percent of prefix origin pairs that are RPKI protected, 3.2 percent are identified as invalid.

Next slide, please. So quickly, we then go on to look at validation results in the ARIN region. For IPv4, where just one quarter of prefix origin pairs are protected, we’re seeing 1.6 percent of invalid announcements. That could be considered a significant percentage given we only have such partial RPKI coverage in IPv4.

And for IPv6, again, the level of RPKI coverage and invalids can be said to be on par with global stats, with 3.1 percent invalid prefix origin pairs observed in the ARIN region.

Next slide, please. So now my colleague Mark will show you in a second the subregional trends that we’ve identified in the ARIN region. Here I just briefly wanted to mention that our platform allows you to view both indicators, so coverage and validation results per country within the ARIN region. So anyone that signs up to our platform can sort of sort through this data.

Next slide, please. And I will also show that the data analytics platform allows users to perform personalized queries. And here’s that sort of snapshot of the interface that essentially you could do, for example, an ASN lookup or prefix lookup and access specific information about validation results, countries impacted, size of IP ranges, RIR impacted, and so forth.

So, again, the platform is available to anyone in the ARIN community looking to take a closer look at the data.

So with that, I’ll pass on to my colleague Mark, who will continue on with the presentation from Tampa. Mark, please take it away.

Mark McFadden: Thanks Carolina, and for those people who are online I am not in Barcelona.

The last two slides that Carolina showed are what makes this community grant project interesting. It means that the RPKI data can be queried in ways that, for instance, the NIST data cannot be. We can do queries based on ASN, based on the prefix, and also on the country.

And I want to talk a little bit about what we found in terms of deployment in the ARIN region and some trends. I also want to talk to you a little bit about what we found about invalids, which we think is pretty interesting. And then I’m going to give Brad a compliment.

(Laughter.)

So let me say a few things about some subregional trends. ARIN is a very diverse community. And so there are very large economies. There are small economies and there are tiny economies.

In the Caribbean region, and yesterday we had a wonderful introduction to the Caribbean region and the issues going on there, one of the things we see for RPKI in the region is four distinct patterns of deployment.

First of all, there are countries in which there is significant deployment of RPKI. We’re defining that as greater than 50 percent. And then there’s a small set of countries that have moderate deployment, at 20 to 50 percent, and then as you can imagine some with little deployment and some with no deployment. This is pretty much IPv4 specific and I’ll talk about that in a minute.

The little graph that you can see here sort of shows you the distribution of RPKI deployment in the Caribbean region. And this is grouped, again, if you go to the data itself you can see it on a country-by-country basis. And in fact, two slides ago Carolina showed how you can use the tool to actually look at each country’s deployment.

Intriguingly, the big difference here for IPv6 deployment are that those in the bottom two groups have no deployment of RPKI for IPv6. So to say that in another word, what we see are countries in the Caribbean region that are adopting RPKI and doing a good job with it and doing that primarily for IPv4.

In the Caribbean region, one of the things we found is that the number of invalids is tiny, it’s tiny. In fact, it can be measured with the fingers of two hands. And there’s a reason for this, of course, is that the number of routes covered is actually very small, right? And the pattern of deployment suggests that the only time that we see invalids in the Caribbean region are because of configuration problems. There’s no obvious sort of hijacking or attacks underway.

In the Caribbean regions that are served by multiple ISPs, for instance, I’ll just pick on one, Puerto Rico, see invalids for isolated routes, and I’ll show you an example of that in a second.

This is a slide that I found really interesting when I discovered the data. First of all, Canada has about, a little bit more than 30 percent of its routes have valid BRPs. And it’s 50 percent for IPv6. And by the way, that’s a pattern that we see in the United States as well.

And the invalids are tiny, less than 1 percent. But what I expected to see is that the prefixes that were protected would range from, say, a /24 to maybe a /16, and that the number of those prefixes would be consistent with the size of the prefix. I was wrong.

What this shows is that the prefixes that are actually most often protected are the /22s and then come the /24s. And if you look at the X axis, you’ll see that it isn’t in order.

What’s getting protected here are different prefix sizes, and once you get up to even a /18 it’s not in the order that you might expect. This is a trend that we found in other countries as well.

In the United States, a little bit less than 25 percent of the routes are valid VRPs. And it turns out that in the United States there’s really good deployment for IPv6. Invalids are around 2 percent, and that’s pretty impressive considering the number of VRPs. And Brad talked about this.

I want to stress something now. Remember that Brad’s data and this data and the NIST data are all coming from different data sources. I’ll talk about that in a moment when I get to the methodology.

But it’s much, much more common in the United States to have multiple invalids for a single AS. And so that’s an interesting discovery. And one of the things that Brad and the ARIN team have done is improved the tool set that supports deployment. And it’s clear that part of that work also needs education and outreach, I think. In the United States, the protected prefix sizes are even a greater range of prefixes than we saw in Canada that go all the way up to /12s.

Now what about invalids? I’m going to show you three examples here. We saw patterns emerge in the data and we were really interested in this, and this might actually help ARIN in terms of how it configures its tool set, which it’s working on, and by the way, the research team thinks is excellent.

But it also tells you about patterns and deployment. And so I’m going to take these three patterns, which we’ve seen throughout the data. It’s really intriguing that these patterns emerge in multiple countries in multiple places.

So, let me take the first one. This is an example in the British Virgin Islands. And let me stop myself here. We’re not protecting RFC1918 space here. TWhat these have been is anonymized for the purposes of the presentation.

If you’re actually interested in this you can actually go find the data on our platform. But what I’ve basically done is tried to illustrate for you that there are three consecutive prefixes there. And, of course, they’re given there as examples.

But those prefixes all belong to a single ASN. However, for the first /24, one VRP covers the route prefix, but no ASN matches the route origin ASN, so it’s invalid. And it looks to us, the research team anyway, like a configuration error and not an attack or not a vulnerability of any kind.

We see this pattern emerge many times in the data. And it’s not just in the Caribbean. It’s in other places as well where what you’ll see is a group of connected /24s that make up a single ASN are trying to be protected, but one of the prefixes is misconfigured. We see this repeatedly in the data.

Another example, which is very similar, is we’ll see, once again a set of /24s, but each one of these goes to a different ASN. And we saw this several times in the data where in the case one of these /24s is going to be invalid for some reason. And in this case the second to last bullet actually tells you why.

But in this case we did some further digging looking through Whois and other sources and we found that the allocation of all four ranges was to someone who is participating in the ARIN transfer program and using the ARIN transfer policy.

And so what we suspect here is that this is just a leftover configuration as it moved out of a production space into the transfer space and to the broker who is actually managing the IP transfer.

Another case that took place in Canada is that we see ISPs configuring one VRP for every /24 they have. However, in this case, it’s a completely different problem. But one of the things that will happen is, isolated, out of nowhere, one of these prefixes will be invalid. All the rest of them will be perfectly valid and will work.

And this is a case where we see this happening a lot. It appears that an operator, an ISP, is configuring the /24s and somehow gets one of them wrong. And this particular case is from Canada. Again, the data is anonymized here, and I’m not providing the ASN that’s associated with it.

Let me say a few words about methodology, and then, I can’t believe this, Hollis, but I’ll be on time here.

One of the things that’s really important about this, one of the things, one of the reasons I really believe in the ARIN Community Grant activity, is because it gave us a chance to do something a little different. NIST has done some outstanding work of doing long -term longitudinal studies on RPKI deployment. But what we were really interested in was localized deployment in the ARIN region.

So we wanted to find out, down to the AS number, who was actually deploying, what their success rate was, and whether or not there were any patterns.

So what we used was route views for the raw BGP data. Now we used six vantage points. That gives us about a 94 percent coverage or so of the global routing table. It was enough, we thought, to cover the ARIN region.

We’re considering in next steps, and Carolina will talk about this in a moment, we’re considering expanding the number of vantage points. For instance, the NIST work uses a larger number of vantage points, but doesn’t allow drilldowns into countries or AS numbers or prefixes and so forth.

We use Routinator to actually study the Route Origin Validation. That, again, is a publicly available tool. It seems to us to be excellent. And then we use the RIR Public Stats Files for geoinformation. This is the mechanism that we’re using to try to find out information about where our VRP should be assigned to, what country we should assign it to.

We’ve been consistently cross-referencing our work with the NIST work that’s been going on and the MANRS data to assess the results. And we think we’re in good shape. Our data is very similar to the NIST data, and that’s great because that confirms our results in terms of the per-country data.

We’re continuing to finesse those algorithms. I can say specifically that we’re not yet satisfied, and Carolina may say a word about this, we’re not yet satisfied with our work on with IPv6. The patterns that I showed you here are all IPv4 patterns. And perhaps the next time I talk to you I’ll talk to you about what we found in IPv6.

We also had a thought here. The other studies, for instance, the MANRS study and the ongoing NIST study, all use as their unit of measurement source-destination address pairs protected by VRP. So basically what you have is a route and an AS number, right, and it’s protected. And that, in our study, we kept that.

But one of the things we became interested in was the number of IP addresses being protected. So instead of protecting the routes, one of the things that RPKI does is actually not just protect the routes but protect the endpoints that are on either ends of those routes from hijacking and other kinds of risks.

And we think that another interesting measurement might be the total number of IP addresses served by routes protected by a VRP. So rather than count them one by one, count them based on the size of their prefixes and then add up the total number of IP addresses.

Fortunately the data that was collected in this project supports this metric. And I’m interested in hearing if people think that’s an interesting new way to look at how well RPKI is being deployed.

With that, I’ll turn it over to Carolina, who is in Barcelona.

Carolina Caeiro: I am indeed in Barcelona.

So next slide. So I’ll go very quickly through next steps, so as Mark explained, we’re finalizing our data analysis. We’re seeking to continue to improve our work and analysis on IPv6. We’ll also continue to finesse our algorithms. We’re looking into expanding the number of vantage points that we’re considering.

And after we go through that process, we’ll be launching our online report with live indicators on the DNS Research Federation website, which, again, will be available for continuous consultation by members of the community.

Mark and I will also be submitting a blog article to ARIN outlining some of the trends we presented today and our assessment of RPKI uptake in the ARIN region to date.

And lastly we’ll be working on getting the word out about the work we’ve been doing. We’ll be submitting presentation proposals to NANOG and CaribNOG, in particular, where we want to present some of this data. So hopefully you’ll see us in some of those additional spaces.

And to wrap up again, as a call to action, if anyone is interested in having a closer look at the data, signing up to our data analytics platform, DAP.live, which also contains many other interesting data feeds, we’re happy to get you set up. So please do get in touch. You have our contact information here on the screen.

So with that, we’ll wrap up and open up for questions.

Hollis Kara: Wonderful. The queue’s now open for any questions for Mark or Carolina. I see the far microphone.

Kevin Blumberg: Kevin Blumberg, The Wire. So the first thing you mentioned was some of the error situations. A great example of that is when using the transfer market, and this may be good for ARIN staff in terms of adding in some notice about it, when you transfer out the space that whole block gets removed as an RPKI valid, you have to reroll it. And it’s probably that time lag that goes on or just people forgetting at that point, because one person did the RPKI and another person is responsible for the transfer out of space.

Mark McFadden: That’s really helpful to know, that’s really helpful to know. And it is a pattern that we see in a variety of, in a variety of places in the data.

So it’s not just confined, for instance, to the United States or Canada. We see it everywhere. And it sounds like it’s just that exact situation.

Kevin Blumberg: So the auto reroll feature that ARIN is now introducing…

Mark McFadden: That Brad talked about will be great.

Kevin Blumberg: …will probably help with that.

The second thing was ARIN statistics, when they’ve been throwing them up over all these years, have used a /24 equivalent for v4 and /48 for v6. So instead of per IP address the number of protected, just the number of /24 equivalents is something that we’re very familiar with in the community.

And lastly, how live is your data?

Mark McFadden: Friday

Kevin Blumberg: Okay. So the one thing that I’ve seen, and to point out, I personally use a tool, IRR explorer. I want to see a net block or an AS. I want to see how valid the IRR and the RPKI is, I use RIPE stats, all these tools.

And all of them have a time lag. And I think that time lag is part of the danger with these studies because, as an end user, I don’t know how fresh my data is. And is it actually valid or was it valid 12 hours ago and now it’s not valid?

So there’s this time lag with the real time stats reporting of the views that I’m seeing. And any ability to get closer to real time with that, I think, would be very good for the industry as a whole.

Mark McFadden: Thank you for all that. That’s really helpful. Let me make a comment about the timeliness of the data. And not to give you another tool to experiment with, but do take a look at DAP.live. When we publish the data we actually say, we make it clear when the data is from.

But, for us, to actually process the data from those three data sources takes us in the neighborhood of 24 hours. There’s a built-in time lag that we can’t do much about.

And then processing, reprocessing the data is something that we haven’t committed to a regular cycle yet.

Kevin Blumberg: Thank you.

Andrew Dul: Andrew Dul. Thank you very much, both of you, for this talk and the work that you’ve been doing. I think this is really valuable for the community. And to the ARIN grant folks who approved this, well done. This was excellent use of the members’ money.

In terms of your question about address percentage of averages covered, I would echo Kevin’s comment about equivalents, or also I do like the just percentage of total allocations covered, because I think that allows us to see how close we are to getting to the magic 80 percent, or whatever number is, when almost all of our actual endpoints are covered by ROAs.

With regard to the freshness of the data, tools are often helpful when you can if I’m using a tool to work for a particular client, I’m usually focused on an AS.

So the ability to perhaps regenerate or look at more recent data for a specific AS, say I want to look at this AS and if you come back in five minutes, I’ll churn up all the data and give you a fresh view, I don’t need to see the whole world.

But, you know, I’m often looking at a per-AS basis, specific, you know, how are we doing for a specific AS.

I think that’s all my comments, so thanks again.

Mark McFadden: Thanks for that. And I will take very seriously the idea of /24 and /48 equivalents. That makes perfect sense to me.

In terms of selectively regenerating the data, I’ll take that back to the technical team and sort of say, is that a possibility? That certainly would cut down on the amount of time it takes to regenerate the data.

Andrew Dul: Do you plan to continue this as an ongoing offering for the community, and are you funded to do such a thing?

Mark McFadden: I’m pleased to tell you you’ll be getting your answer from Barcelona.

Carolina Caeiro: Thank you, Mark. Yes, we are planning to offer this in a sort of continuous way. And we are funded by the ARIN Grants Program for just the one year. But we have our platform, which contains, as I mentioned before, multiple other data feeds as well.

And it basically, there’s the elasticity cost with us, sort of maintaining these databases up to date, so we’re planning on doing that. So you have, you’ll be able to access the data feeds on the DAP.live platform, and then you’ll have sort of the report, online report version of it on our website that we’ll also be updating and showing visualizations pulling data from the DAP.live system.

Andrew Dul: Thank you very much.

Hollis Kara: One more at the far mic.

Louis DeVictoria: Lou DeVictoria, Perimeter81 and a fellow as well. So earlier in the presentation it was demonstrated that globally RPKI for, globally, was about 50 percent. And in the ARIN region it was about 25 percent for IPv4. What conclusions, if any, did you understand why the ARIN region is so far behind globally?

Mark McFadden: So I’m with Brad on the analysis of this. First of all, I don’t really think that ARIN is that far behind. And I think the reason is is that in the ARIN region, it’s important to consider the eligibility of particular routes for protection.

And so if you look a little more closely at the eligible routes, the numbers come more in line with the global analysis that Carolina presented. It’s still lower. It’s still lower. But I think what ARIN is doing and this is not plugging my funders here, but it will sound like it, I think improving the tool set for RPKI and information and education, especially to nontechnical people who are going to take advantage of the tool, those are the things that will dramatically change numbers in ARIN.

Louis DeVictoria: Thank you.

John Sweeting: John Sweeting with ARIN. Just a, so a quick data point to that, 32 percent of our IPs are not eligible for RPKI, 32 percent.

Mark McFadden: That’s a huge number. And that will be reflected in our written work.

Mike Burns: Mike Burns, IPTrading. Yeah, it would behoove us to get legacy holders to be able to do RPKI easily.

My question is can you query your data for ROAs that, where the registrant of the prefix is different from the registrant of the ASN?

Mark McFadden: What a great question. And the answer regrettably is no. But let me take that back. That’s something that, just you’re saying it out loud, interests me, suddenly. And so let me take that back to the tech team and think about whether or not that’s something that — we do have the data to make that comparison. So I know we have the data.

And do we have the ability to actually do that in the query engine that we already have set up, or would we have to create something new? That’s what I don’t know the answer to.

Mike Burns: The reason I ask is it’s a proxy for leased addresses, and I’d like to see if RPKI is as high as I expect in that community.

Mark McFadden: That makes total sense to me.

John Sweeting: John Sweeting with ARIN. Mark, I have a question, too, back to you were talking about transfers and you think maybe it’s because there’s two different people trying to put ROAs out there.

I think that’s a little bit impossible, at least in ARIN, because we strip all that. Before we transfer, all the RPKI, all the ROAs are stripped. And nobody else can put ROAs on those until they’re actually in their accounts.

Mark McFadden: So I think the way I should address it is maybe have a chat with you offline, because the patterns show up in the data over and over again, where…

John Sweeting: Yeah, it would be good to take a deeper look at it. Get Brad involved.

Hollis Kara: Wonderful. I think that’s everything. Thank you so much for a great presentation.

(Applause.)

I need to allow just a moment for us to transition between virtual presenters, so I’m going to stick with the RPKI theme and ask Brad, would you mind coming to the microphone, because there was a question in chat, and I think you can probably give a pretty quick answer to it while we’re giving them a moment to set up. I tried to Slack you, but you have your notifications turned off. Dude, really?

Brad Gorman: I’m busy at the ARIN meeting.

Hollis Kara: Yeah, so am I, man. I’ll ask my question while you’re walking. Basically a question came up in chat, if you can explain really briefly the benefit, difference between hosted and delegated.

Brad Gorman: Very briefly.

Hollis Kara: Very briefly. The elevator version.

Brad Gorman: You have many fewer responsibilities put on you as the organization who chooses to use RPKI if you’re doing hosted over delegated.

A delegated customer probably has more resources internally to be able to run the systems and maintain time and host the equipment that the vast majority of organizations don’t have.

Hollis Kara: Awesome. Hopefully that was, was that a good stall? Do you need more time? I’m not tapdancing. Nobody wants that. All right. We’re good. Wonderful.

We’re going to move into our next grant report. We’ve got our virtual presenter queued up. Harlan Stenn is going to give an update on Network Time. And I’m going to put the clicker down and go back to my seat.

Network Time

Harlan Stenn: There we are, and everything seems to be working.

Hello and thank you. I’m here to talk about our grant, which was to help the NTP Project clean up an extension field portion of the NTP packet to allow us to implement Network Time security.

Next slide, please. So this is just a brief history of NTP. NTP3 ended about 1998. And the first version of NTP4, an alpha version, was released right about ‘98. I don’t think it was into ‘99.

And you can see that this was out there that had Autokey, the only, at the time, consumer of NTP extension fields happened then. And it was quickly taken up by a number of people who wanted to use it.

Somewhere in the mid ’90s, before this happened, a few organizations were talking with Professor Mills about their needs around ephemeral public key exchange for NTP packet authentication.

And they came up with a designed and engineered solution to provide this. And it was for very specific use cases. And that’s what got put into the code.

And due to the way the standard was handled, at least back then, we wanted to make sure that everything worked right before we put it into a standard.

So the first versions of extension fields worked pretty much out of the box. The first user of an Autokey had a life of probably nine years before the very first version of the Autokey draft came out in September of 2007.

Next slide, please. In 2009, we were already up to NTP4.2.6. 2010 saw the actual publication of the NTP version for RFCs, and that was one for protocol and algorithms and a second one for Autokey. So it had been about 12 years that all this had been in place.

And one of the things — okay, there we go. Q3 of ‘97 was the very first time NTP Version 4 code came out. And the first draft standard was 2005, but the actual publication wasn’t until 2010. So there was a long period of time there were, people were actually using our code.

And next slide, please. The only difference between NTP Version 3 and NTP Version 4 at the protocol level was that we created an extension field at the end of the packet to allow us to do more than just basic timekeeping.

And the only user, as I’ve hinted at before, for extension fields from 1997 until 2015 was the Autokey protocol. Several other extension fields started to be developed and worked on in 2015.

Next slide, please. The NTP Version 1 extension field that we implemented is shown in the top diagram. And I only hope you can see that well enough.

The left-hand bits have a couple of flag bits and then, I have to make this bigger so I can read it, it looks like the next thing, the next major field was a type followed by a code. And the standard went ahead and switched the type and field codes around.It was a little tiny bit more to it than that, but that was enough to see what’s going on there.

And changing that is not as simple as simply switching the order of them, because at this point, what we discovered is since the Version 1 and Version 2 extension field headers got added very late in the game, while people were using it, Dave took the approach of, we weren’t going to change it in the distributed code until somebody actually needed it because there were active users, and Dave didn’t want to break backward compatibility.

Next slide, please. So as I mentioned, the code was deployed using the Version 1 extension field structure and they changed it. So, you know, we got bit.

Next slide please. Now that we do have Network Time Security in the NTP protocol spec, the NTP Working Group — NTS is basically the working group’s design of an ephemeral keying system. And when the new players implemented NTS, that’s when it came up and bit us that our implementation needed to support the V ersion 2 extension field spec so we could easily interoperate.

And in spite of us asking people to stop using Autokey probably back around 2011 or ‘12, there are people who are still using it for their own internal, apparently, good reasons. And so we need to be able to support people who want to run both the version 2 extension field format and the version 1.

So I attended my first NANOG last fall in Los Angeles, and I overheard a bunch of folks talking about the IETF community and the NANOG community. And what I heard was that the primary reason IPv6 looks the way it does and has the deployment issues that we’ve seen is that the NANOG folks believe they’re all about the operational issues, while the IETF is more about what I’ll call political solutions and consensus.

And that may or may not be an accurate assessment of what’s going on, but I will say that it mirrors my experience with the NTP Project and specifying things through IETF.

The NTP Project has always been about the operational and engineering issues. And in my experience, the IETF NTP Working Group is primarily now driven by consensus and will prefer consensus over technical and operational qualities of their decisions.

I hope it was okay for me to say that. Next slide, please.

So we’ve got the grant and have started working on the code to convert everything to support both extension field formats. And we’re about halfway through that code conversion now.

These changes affect many lines of code and several of the longer code files. I think we’re probably talking about some files that have thousands of lines of code in them. And we have also had to add some new files to the codebase.

And we still need to specify the changes that we’re going to need to the NTP configuration file to make sure that the association you’re using gets the extension field format that you expect for that.

That’s not a particularly big deal. It’s more of a “what color do we paint it” sort of thing. And we are on track with the implementation of this thing as far as the contract goes.

Next slide, please. So to conclude, yes, we’re incredibly appreciative of getting the community grant from ARIN because we are resource starved here, and this allowed us to actually make progress to get our implementation to include NTS in our NTP implementation.

And that’s what I got.

Hollis Kara: Wonderful, thank you. Did anybody have any questions for Harlan?

Louis DeVictoria: Lou DeVictoria, Perimeter81, Fellow. Just a question. Is there RFC or a draft or something we could see on the work in progress?

Harlan Stenn: Is there a what?

Louis DeVictoria: Is there a draft or an RFC, draft of progress of the work that’s being done?

Harlan Stenn: There isn’t an RFC because what got originally passed was the version 2 format, and we implemented the version 1 format.

So I don’t have a graph or a pretty picture because my attempt to do that was to say we’re about halfway through the coding effort. So we know what has to be touched and where. I know where we are, so we’ re happy that way.

Louis DeVictoria: Fair enough. Thank you.

Kevin Blumberg: Kevin Blumberg, the Toronto Internet Exchange. We run a public NTP stratum 1 time service, one of the few NTP stratum 1 public services. A lot of them have closed down or gotten very specific.

It’s great that this work is being done, but my concern is scalability and performance testing afterwards because at scale, just a simple addition of any level of encryption can have a detrimental effect to the processing power that’s required on our side.

Is that something that you’re sort of looking at as far as an optimization to make sure or to at least be able to have here’s without NTS, here’s with NTS in our code, to give operators sort of at least a head start on what scaling issues they may have?

Harlan Stenn: Awesome question, and I’m going to be careful with my words here. I mentioned the difference — I’m not going to be that careful with my words here.

(Laughter.)

I mentioned that the IETF can occasionally be more consensus driven than technically or what I’ll say, call reality driven, and that is being unkind, and for that I apologize, but I also, this is my prime sleeping hours right now, so I woke up early to be here.

We’re implementing NTS because it’s the standard and that’s what everybody wants. And, yes, it is operationally significant that the level of work required for NTS is significant and it does affect the throughput you’re going to be able to get with it.

That said, nothing is stopping anybody from using the private key mechanisms that are already in NTP. It’s just a pain to deal with, and nobody wants to mess with it for large deployments.

I will say that one of the items on our plate internally in both Network Time Foundation and the NTP Project is what I’m calling secure network time.

That’s based on a new process we’re working on called NTPTCP services. And that, we have a specification for an alternative ephemeral keying technique that uses public key technology that we’re pretty sure, without having any code written yet, is going to be more efficient to implement than NTS.

And once we’re ready, we’ll put the thing out there and give it a shot and see if people like it.

Kevin Blumberg: Thank you. And there’s nothing wrong with a little bit of candor when it comes to the differences in the communities.

Harlan Stenn: Yeah, apparently I suffer from too much candor.

(Laughter.)

Hollis Kara: Thank you very much, Harlan, for your presentation.

Harlan Stenn: Thank you.

(Applause.)

All right. We’re going to do one more grant report before we take our break. I’ve got Matt Pounsett from DNS-OARC. He has provided a recording, so we’re going to play his presentation. But he is available virtually, and we will unmic him when we get to Q&A if there are any questions for Matt.

So with that, are we ready to roll tape? We are ready.

DNS-OARC Systems Engineering

Matt Pounsett: Hi, everyone. My name is Matt Pounsett. I’m a systems engineer at DNS-OARC. For those who may not know who we are, DNS-OARC is the DNS Operations, Analysis, and Research Center.

It’s an organization of DNS operators and software vendors, researchers, occasionally security people, all with sort of an interest in operating the DNS.

We run mailing lists, we run workshops a couple times a year. As an organization, most of what we do is facilitate communication and interoperation between different DNS organizations and run some data collections a couple times a year and also do other things in the industry to sort of support the DNS running testing tools and that sort of thing.

So today I’m talking about a project we’re working on which is being funded by ARIN, which is sort of one those side DNS-supporting projects, which is the AS112 community project.

But this is the individual project that ARIN is supporting here is actually a redesign of sort of the information site about the AS112 project, which is now going for a couple of decades now.

So just a quick rundown on what I’m going to talk about here. I’m going to talk a bit about what AS112 is, why it exists, and then the project we’re working on that ARIN is actually funding right now, and some examples of what we’re looking at, and a bit of an information project that I needed to do on the side in order to complete the work.

So for most of you in the room who are familiar with DNS, I’m sorry about this, I’m going to try and go through this as quickly as possible, but in order to understand what the project, what AS112 does, people need to have at least a basic understanding of what’s going on in the DNS world.

A lot of you have probably seen a slide like this at some point before. Somebody at their computer goes to a website, sends an email, whatever, their computer needs to resolve a domain name.

And that query gets sent out to a local DNS server. That local recursive DNS server will send a query to the roots and get delegation down to the next level in the DNS tree, which will pass another, may pass another delegation, and so on, until you get to the nameserver that can actually answer the question. And this usually goes by in a tenth of a second, that sort of thing.

This happens with reverse DNS as well. So looking up, you can look up an IP address and get back a name. And so exactly the same sort of thing happens. They go out to the root servers. In this case, instead of delegating to sort of a named TLD, like .com or net or something, for reverse DNS it gets delegated to inaddr.arpa, which will delegate to an RIR, which will then delegate to whoever is providing reverse DNS for that address block.

So that’s basically how this is supposed to work.

So AS112 is this project that was started a while back. In 1996, people are probably aware, RFC 1918 delegated some address blocks as private address space. And since then these addresses are not meant to be used on the public Internet, and therefore you would not expect to see DNS queries for the reverse DNS blocks on the public Internet anywhere.

But they continued to come. And the reason essentially is that people did not by default configure their local recursive DNS server to either be authoritative for these or to not send the queries out.

So AS112 is this project that was designed to deal with that. So before AS112 existed, you would get something like this, where somebody’s computer might want to look up something in, you know, 10. the recursive server would send in a query to root servers, root servers would send back a nonexistent domain response.

At this time in the Internet’s history, though, negative answers were only optionally cached by recursive servers. So if that recursive server needed to look up that same address 20 seconds later, it would send the same query back to the root servers 20 seconds later, and over and over and over.

And you can imagine what happens if you’ve got hundreds of thousands or maybe millions of recursive servers on the Internet all sending queries every time they need to look up the reverse name for an address or think they need to look it up.

And it wasn’t until 1998 when the DNS spec was changed so that negative caching could work, which was essentially caching a negative response, caching a “nope, this doesn’t exist” answer. And then after that there’s sort of a long tail of deploying new DNS servers.

So there was a long time where these queries would keep coming to the root servers, andthey accounted for a huge percentage of the garbage queries that root servers see.

So AS112 was created in order to deal with this problem. And essentially what that is, is the first step, set up some DNS servers that can actually answer for these RFC 1918 names.

So this now looks more like a normal DNS query, and the nonexistent domain responses or, rather, the delegation responses from the root servers and in addr.arpa servers are both cacheable. So they stopped seeing queries quite as often about these things, and the queries go to the AS112 servers instead, and then they can handle those garbage queries, essentially.

And by the time the AS112 project was set up, negative caching had become more of a thing, and the negative cache on the zone, or, rather, these zones, is one week. So this causes recursive servers to cache these for longer.

This is another thing you couldn’t really do at the root because the root doesn’t want to set quite so long in negative cache.

So it’s a bit of a conflict there between wanting to have new names show up in the DNS relatively quickly but then have things like garbage queries like this get cached, or the negative responses for these garbage queries get cached, for a long time. So the AS112 nameservers answer these queries this way.

Now, the next step in this is why this is a special AS. And this gets down to goofing around with routing a little bit. So there’s this thing that I’m, again, sure most people are aware of, is the idea of Anycast, that you can have nameservers all over the Internet that answer queries for the same things.

So this takes advantage of a thing in Internet routing where there’s essentially no difference between one very well connected network and a whole bunch of completely independent networks all connected to different transit providers or Internet exchange points. The routing doesn’t know the difference. It can’t tell.

And so what the AS112 project became is this loosely coordinated, very loosely coordinated, I’ll get to that in a minute, community -operated Anycast network of DNS servers.

It acts as this sink for DNS queries that should not ever hit the public Internet. Originally answering for the RFC 1918 blocks, but when RFC 6890 was published in , I’m trying to remember when that was, the early 2000s, I think, the link -local addresses were added.

And then in 2015, there was this empty.as112.arpa zone that was added, which supports adding new zones to this using D name, which is a thing like C names, like aliases in the DNS, except it applies to entire DNS trees and not just a single name.

So if you put in a D name for, I don’t know, home, for example, that would mean anything under home gets sent to this alias rather than just a particular, just one domain name.

And so getting on to what we’re actually doing right now. So DNS-OARC manages the AS112 Project, as much as any individual or organization can manage it. As I mentioned, it’s a very loosely coordinated AS. It’s a community-operated Autonomous System.

Anyone on the Internet can set up an AS112 instance. There are instructions, both on the AS112 website and in RFCs, about how to set it up and what you need to do in order to support it.

And the idea is that hundreds or thousands of networks on the Internet could all set up and sync just a little tiny fraction each of these bad queries.

So DNS-OARC operates or is, rather, is the contact for the AS number and the other numbers resources, the addresses involved. We maintain the registration of the relevant domain name. We run an operators mailing list. We run a website with information about the project and how to get set up and how to participate.

And the project we’re working on right now is cleaning up that website. It is a fantastic example of 1995 web design. It is not particularly well-organized. It has grown organically over the years with little bits of information being added here and there and never really, the way to get at particular bits of information doesn’t seem to have ever really been thought about. It can be very hard to find specific things that you’re looking for on the website.

And there is an operator listing, which is another thing that we try and manage there on that list. It is always been best effort in terms of there’s no requirement that anybody add themselves to the operator listing. But, even so, it’s still fairly difficult to manage because of the way it has grown up in this sort of organic way.

So this is what the main webpage looks like today. It’s tough to read. I’m not expecting people in the room to really be able to read this slide, but it’s just sort of meant to show an example of what the site looks like currently.

But if you go to the website, you can see there is out-of-date news. It’s not particularly well-organized. You’d be hard pressed to find it, but there is information in there about how to set up an AS112 node for different systems. And the operator listing I mentioned, which I’ll get to in a moment.

So this project is to clean the site up. This is essentially what we’re doing now. So the new site that I’m in the process of building is better organized. We’ve updated a lot of the information, rewritten instructions to be clearer, rewritten the project description to be clearer.

It’s being set up to be better reactive web development so it shows up on different devices as well. And just generally easier to interact with. It’ll be information indexed better from web crawlers and the like.

So this operator listing, which is just some hand-edited HTML, is actually going to be the major bulk of the work here. Most of what we’re doing is really simple, just like cleaning up the website. And it’s not a big website. That wouldn’t take long.

This operator listing is a bigger project, because it is a listing of a few hundred AS112 operators that has been collected over the decades. It’s all hand-edited HTML. It’s not particularly consistently structured. So it’s not machine readable at all. It’s been edited by half a dozen different people over the years, all with different HTML styles and different ways of organizing things.

There are huge gaps in information. You may be able to see on the slide here there are a few question marks in just the section I’ve shown here.

And there’s no way to do garbage collection on this list, soif somebody takes down an AS112 node, we have no way of finding out about it unless they think to email us.

So there’s no distinction here between globally reachable AS112 instances and ones that are only at ISPs, and there is no consideration for any operators who might have more than one instance. There are a few places in here where somebody’s tried to cram multiple instances into the table, and it’s really hard to figure out what’s going on there.

So we’re working on a new setup for this where the whole thing is going to be converted over to structured data. So the global instances will be stored locally as structured data in something like YAML or JSON.

The listing itself will be structured as well as the data behind it to handle organizations that have multiple instances. For example, Cloudflare just joined the project in the last year and has put up, my understanding is, at least dozens, if not hundreds, of instances themselves. So, and we need a way to be able to accommodate that in the listing.

And another thing that we’re doing is the instances that are only available at IXPs, we’ll be pulling from PeeringDB, using their API. They have a very flexible API that’s used to receive information about what AS numbers are available at individual IXPs.

And we’ve got our AS112 listing set up so that any IXP operator can update our listing. And so anyone who puts up an AS112 instance at any IXP doesn’t have to tell us about it. Their IXP operator can just automatically upload that information to PeeringDB and we can automatically download it via the PeeringDB API and add it to the operator listing.

The other big part of this part of the cleanup is actually being able to clean up that data. And so one of the things that I’m doing with that is we had some help from RIPE NCC using their Atlas measurement network.

You may be able to see on this slide a lookup for hostname.as112.net, which is a special hostname that every AS112 DNS server responds to, which contains information about that specific AS112 instance.

So if you look that up, if you look up that name from your computer, you will find out which AS112 — you’ll find out some information about which AS112 instance is providin g information to your area of the Internet, to your network.

So what I did is I set up a RIPE Atlas measurement that would try and query from as many networks as possible for this information. Being a lowly regular Internet user, I can only send queries using about a thousand Atlas probes at a time, which makes it difficult to be sure that you’re getting a clear view of the entire Internet.

However, RIPE was kind enough to help me out with it. And the staff at RIPE Labs are able to send queries out through all of the Atlas probes all at the same time.

So I’ve got a bulk of data here from a little over 6,000 Atlas probes which have all this information, which I can then process, produce a unique list of responses out of all of these responses. So I’ll narrow it down to probably a few hundred individual answers that are distinct from each other.

And then essentially by hand I can go through that listing and compare it against our old handwritten operator listing and see what matches up and what doesn’t.

So this will help me take the unstructured data and convert it to structured data, which is going to be a bit of a data entry job. It’s going to take a little while. But theoretically we only have to do this once.

And then once we’ve got that structured data built, all of the future entries can be added by a submission form, which is essentially moderated.

So people who are wanting to set up a new AS112 instance can go to the AS112 website and post, fill out a form that contains information about their organization and what AS they’re going to be transiting AS112 through and the physical location and things like that, if they have multiple nodes, where they’re going to be.

And then that information can come to us, we can look it over, make sure it’s not spam or whatever, and then automatically add it to the listing. So there will no longer need to be anybody sort of hand-editing files in order to manage that.

And then one of the things that although it’s not part of the specific project, one of the things we hope that will enable is in the future being able to do garbage collection on the list.

So with more structured data and more complete data, we’ll be able to do things like individually query AS112 instances periodically to see if they’re still responding, match up listings that we have for transit from information we get from PeeringDB to find instances that are available, individual instances that are available both ways, and mark those appropriately, all these kinds of things that this cleanup and reorganization and converting to structured data, all these things that this will enable in the future.

So this is the project that we’re doing. The current status here is we’re about halfway through. We’ve gotthe website redesigned. We’ve got a lot of the documentation rewritten. And essentially we’re working on this structured data component at the moment.

So I don’t have a lot to show for that at this point in time. But we’re expecting to have that done over the next couple of months, deploy it this summer sometime, and be ready for people to interact with and hopefully make it easier for new people to join the project and help out as well as for existing operators to be able to look for updated information or any news about the project or anything like that.

So if there are any questions, I’m expecting to be available here on the, remotely, on ARIN Zoom call. If anyone would like to ask anything about the project, about what we’re doing, I’d be glad to answer those questions now. Thanks very much.

Hollis Kara: All right. Briefly, before we go to break, we’re going to bring Matt up on screen. Somewhere. I think. There he is. But he’s frozen.

If anybody has any questions for Matt, if you could please approach the microphone or start typing real quickly.

Oh, gosh, and there he’s gone. Did we lose him?

If you have any questions you would like communicated to Matt, please let me know, and we’ll make sure those reach him.

At this time we’ll take a break. It’s 10:47. Let’s come back at 11:10. We’re a little bit behind schedule. I think we can get ourselves back on track if we restart around then. And we will see you after the break.

(Break from 10:47 AM to 11:11 AM.)

Legacy Fee Cap Expiration

Hollis Kara: Come on in, folks. We need to get back started. Give folks a minute to come in from the hall.

We’re going to get started. We have a few things we want to fit in. We had to re jigger our agenda a bit because we have a hard start at 11:30. We have a number of guests joining us virtually for the keynote. I want to respect their time as well as everyone in the room’s.

So I’m going to ask John Curran to come up to the stage right now and give an update on the Legacy Fee Cap Expiration.

(Applause.)

John Curran: Good morning, everyone. I’m John Curran, CEO of ARIN. I want to talk about a change to our fee practices that’s pretty important. Most people in this room may not be affected, but some will. It’s really just calling to your attention a good thing that’s going away. We’re making things more equitable across the board.

So Legacy Fee Cap Expiration. So we provide Internet registry services to everyone in the ARIN registry, and that includes a number of organizations that have no contract with us, what’s called legacy resources.

And this is a situation where, when ARIN was formed, there was an existing registry. We did not tell everyone you have to enter into a contract. What we said is you’ll get the same services you get, rather than being paid for, now that this is a membership organization and not a US government function, you’ll get the same services, the US government is not paying for them. Our membership will absorb those costs. You don’t need a contract, and you’ll just be carried.

So we have a number of legacy registrants who have been getting services without any fee or without any contract for more than 25 years.

Everyone else, when they join, when they get resources and when they transfer them in to the ARIN registry, they pay fees, the normal fee schedule. And that also — that fees go and pay for the development of advanced services. Example is our authenticated IRR or our RPKI, Resource Public Key Infrastructure services, were paid for by the ARIN members who are paying fees.

So the legacy holders have been getting free services. But when it comes to these advanced services, we haven’t provided them to them. Why? Well, because everyone else is paying for them.

The legacy holders actually have had the ability to get services, all the services, including the advanced ones, and still pay less. Not nothing, but enter a contract, and they pay the same fees as everyone else, but those fees are capped. It’s called the legacy cap.

Okay. If they enter into a Registry Services Agreement with us, they get the advanced services. They’re treated just like every other member. But we cap their fees.

And historically, this has been for the legacy resource holder can bring their resources under contract and have a very attractive price compared to everyone else.

How attractive? If you put your resources under a RSA or LRSA, you get a contract that describes your rights. You get a clear relationship with ARIN. And you’ve got definition of the services, the legal responsibilities and rights.

The maintenance cap right now, $175, and it goes up $25 per year, no more. And so if you bring your resources under an agreement, you get the same service as everyone else. Even if your fees would have been $1,000, $2,000, $4,000 a year, you pay $175.

Now, you’ll pay $200 some other year and $225 the year after, but you’re not paying $4,000 if you bring it under a Registration Services Agreement.

We’ve offered this fee cap to legacy holders for some time. Since the beginning, we’ve said it’s a fee cap, but it won’t go up more than $25 per year.

So the Board has basically said we need to move towards equitable treatment across all our customers. And so the ability to get this fee cap, which is applied to your services and your fees going forward, that will end at the end of this year.

At the end of this year, if you sign a legacy resource services agreement, which is the same as the regular registry agreement, the fee schedule will be the regular fee schedule.

You actually have the same thing now. If you sign now, it’s the regular fee schedule, but we apply the legacy fee cap to your fees now and going forward into the future.

This can make a substantial difference. If you know someone who has a legacy resource, you should have them look one more time at the RSA. We’ve changed it several times. In fact, most recently, we removed a couple of provisions that people have been complaining about for years as saying they get in the way of them entering an RSA.

Okay. We got rid of them. Now look at it one more time, because the fee cap that you will get, if you enter this year, is never going to be available again. And if you enter an RSA this year and you — you’ll have the benefit of the fee cap, and that will affect your fees going forward for all time going forward.

You’ll have that fee cap. If we wait 100 years, you’ll eventually pay probably the same as everyone else because we go up $25 per year. But the fact of the matter is that that’s still substantially less than other customers are paying.

Beginning on January 1st, 2024, there will be no fee cap on legacy maintenance fees for any legacy resources brought under an agreement after December 31st, 2023.

If you have it in place, it continues. But if you don’t have it in place, you can’t get the legacy fee cap starting next year. You’ll pay the same as everyone else.

So very important news. Want everyone to pay attention. If you know a legacy resource holder and you think they might need the advanced services, they want a normal relationship with ARIN, now’s the time to look at that.

Tips for governmental entities. Some folks who want to have these services, well, the problem is that if you’re a government entity, sometimes you can’t enter into a contract because there’s clauses you don’t like.

We actually have done a number of changes. We do amend the RSA/LRSA for government entities for clauses that they’re prohibited to enter in by law. We actually have the ability for them to go online and actually submit the specific issues, and we’ll generate an agreement minus those clauses or with those clauses modified.

And so if you’re a government entity, look online for government support or call the Registration Services desk.

Okay. That’s it. Thank you. I’ll take questions on the removal of the legacy fee cap. Remote questions, on site questions.

Thank you very much.

(Applause.)

Hollis Kara: Thank you, John. Actually, you can just reintroduce yourself.

John Curran: I’m going to come back. I have the next presentation, which is an update on happenings in AFRINIC. I believe that’s being queued up now. There we go. Okay.

Status of AFRINIC

John Curran: So the RIR system is made up of a group of RIRs, five RIRs. And one of them is AFRINIC, and AFRINIC is going through a number of developments. I wanted to bring the ARIN community up to speed on those.

So here’s the five RIRs. ARIN handles North America, Canada, and about half of the Caribbean. LACNIC handles the other half, South America, Mexico. RIPE. AFRINIC handles the African continent, APNIC for the Asia-Pacific Rim.

AFRINIC was established in 2005. Has been doing a great job for many, many, many years.

Because what I’m talking about is matters that border on legal matters, I need to say we would normally not be discussing such activities in another RIR. And I’m going to try to do the best I can under the constraints I’m under to provide some insight and at least some information for people.

At the end of the day, it’s one registry system. So what happens there does affect us, and you need to be aware of developments.

So timeline of the legal developments. In 2020, AFRINIC completed a registry audit. They sought more information from one of their customers, Cloud Innovation, a gentleman named Mr. Lu Heng, the principal behind that.

They made a determination that the resources weren’t being used for the purpose issued, and they said they wished to recover those and revoke those, allowing for, of course, customer migration.

Cloud Innovation disputed the authority of AFRINIC to do this and then sought several legal injunctions in the courts of Mauritius to prevent having to return the blocks to AFRINIC.

One of those, all of this is pretty normal up to this point. What I mean by that is, you all have agreements with RIRs. You have one with ARIN. If you have a dispute, there’s provisions for you to go into litigation, mediation, and a court action to enforce your rights.

You need that. You absolutely need the ability to enforce your rights in a court. That way the agreement has meaning. So disputes with RIRs are normal. They do happen. People should be able to enforce the contracts they have.

But one of the motions that was filed froze AFRINIC’s bank accounts. And makes it kind of hard to pay your vendors, your colo, your communications, your staff if you can’t get access to your bank accounts.

And that began to endanger the overall stability of the system. We actually managed to get the freezing order modified and removed.

And then AFRINIC went into an election cycle and announced an annual election, and it had an irregular structure to it. It wasn’t the list of candidates you might have expected.

An injunction was sought by Larus Cloud Limited, which was associated with Cloud Innovation, granted ex parte by the court, preventing AFRINIC from holding its election. So it was unable to seat new directors.

The challenge is that AFRINIC actually at that point fell below the number of directors to have a Board meeting. And if you can’t have a Board meeting, you can’t have Board resolutions. If you can’t have Board resolutions, you have trouble doing things like adopting a budget and making key decisions and holding elections or calling for a governance meeting.

There was a couple of attempts to remedy this, a filing by registered members of Africa, which are actually the Board members seeking to appoint directors to constitute a quorum to get them back up over that number. That didn’t happen.

There’s ongoing litigation. AFRINIC has many claims. Majority of those involve Cloud Innovation. The financial accounts were frozen for some time, as I noted, and it impacted the operations of AFRINIC. AFRINIC was able to eventually adjust to this.

So AFRINIC’s been going through quite a few challenges. And, in fact, one of these challenges was that in November its CEO, Eddy Kayihura, his contract was, well, it’s either his contract wasn’t renewed or was up. Doesn’t matter either way. The lack of a Board certainly hindered that.

And they were operating under emergency powers, powers defined in the contract, in the bylaws, that allow them to function and allow the CEO to do certain duties when it’s not a full Board.

That’s actually ended as well. That actually happened right after our ARIN Hollywood meeting.

So it’s been a challenging situation. The injunction prevented the annual elections. No quorum before the Board of Directors. And that requires a court to untangle. You can’t readily get out of that situation because you’d have the Board have to take an action, and if you can’t convene the Board, you can’t take an action.

So there’s also a public relations campaign by an organization called NRS. NRS is the Number Resource Society. It’s not at all related to the NRO, which is the coordination function of the five RIRs.

NRS is out there providing information on its views of the openness, the transparency, the governance, the integrity of the registry system. It’s very hard to figure out, however, who the NRS is because it’s a fairly opaque organization itself to the outside.

It also included attacks against Eddy Kayihura and other individuals who have supported AFRINIC.

So we have a challenging time. The thing we need to have is we need to have a set of fair, transparent, and trustworthy elections in AFRINIC to get a full Board in place. That’s our top priority. And, in fact, that activity is ongoing.

The NRO has been doing what it can, the Number Resource Organization, the coordination of the five RIRs, noting that each RIR is set up to be self-governing. At the end of the day, what we want is the AFRINIC Internet community to come together and handle the self-governance challenges of AFRINIC. That’s not something that really the other RIRs can do, but we can provide support and some clarity to the situation.

We actually did send a letter to the Mauritius government noting that the quantity of lawsuits, including the injunction on the bank accounts, could impact the ability of AFRINIC to perform a critical function.

And AFRINIC is not just another not-for-profit, another company, a membership organization operating on a non-for-profit basis. It is providing a critical service to all of Africa.

And we did call that out to the Mauritius government; that the granting of orders ex parte, for example, without the ability to even respond, could make it very difficult to have infrastructure operated in Mauritius if it can be terminated such ways and impacted.

We also sent a letter to ICANN on the 21st of March asking ICANN to help when we get into an election process, help with making sure that those are fair and open and well known so the entire Africa community can participate and be engaged in that process.

Our communications to the community. As I said, current status, there’s significant pending litigation. There’s insufficient directors to hold a meeting to get more directors.

The petition to the court for appointment of temporary directors have not been granted. The bylaws provided emergency powers, which have ended with the departure of Eddy Kayihura.

It’s still running properly. The team in AFRINIC is very dedicated, and the chief financial officer is sort of the acting manager there. He’s doing a wonderful job, along with the engineering team. The legal staff is on site, and they’re handling the responses to these. We actually have fairly frequent communications with them.

And customers are being served. Customers are being served. They’re getting the registry services. It’s just the governing function that has been impacted, and so fairly problematic on that basis.

Next steps to be determined. An order has been put in calling for a Members Meeting and an election of Board members. And that’s made by one of the directors of the Board and also supported by some of the ex-directors of the Board. AFRINIC has a function called the Council of Elders, which is made up of past Board members, and they’re also supporting that effort.

We have to see how the court views that and whether they see it favorably. It calls for an expedited election process to get the Board up to quorum, which is necessary, if you think about it, because when you fall below quorum and you don’t have enough directors, it’s very hard to execute. Even things like an election and appointing a NomCom reference a functioning Board. So it had to actually specify how to get past those hurdles.

Hopefully this will be favorably seen. ARIN has made representations. We’re standing by to provide any support AFRINIC needs, technical, financial. We’ve provided financial support and legal support in the past; we’ll continue to do so.

So we’re hoping for a successful court ruling that would allow AFRINIC to have elections, and hopefully that will let it get back to operating normal. They’ll still have to deal with the litigation, but you even can’t deal with the litigation until you have a governing Board.

So hopefully this is the path towards getting AFRINIC in the clear. And that’s our good scenario, the one we’re all aiming for.

So, with that, I’ll end. I’ll take any questions.

Hervé, yes. Speak slowly and clearly.

Hervé Clement: I will try to. Thanks very much and thank you, John, for this clear presentation because the situation is very complex. And so you succeeded in having something so clear.

But so we have not a clear view on about to solve this governance situation, which is, I can imagine, the first time happening. And so next question, but good question, about the governance and the global governance system.

So I am here. So I came also with my hat of Chair of the ASO AC this year. And one of the consequences of this situation, so with no CEO, with not quorum reached within the executive offices of AFRINIC, if so there is no possibility to appoint any of the member to the ASO AC.

So normally the ASO AC is comprised of 15 members, three from the different regions. We have only one this year, Saul Stein. And so his term finish at the end of this year, so if the situation is not solved, and I have no idea what will happen, we will have so next year only 12 members, and without the AFRINIC representation, which is, in my opinion, perhaps a more important problem. It’s a problem of representativity, so regarding so the governance as a multistakeholder, et cetera.

So you said before that normally it was an internal affair within AFRINIC, but it’s not anymore. So because it affects, as we can see, the ASO AC, the ASO globally, et cetera.

And we had a lot of discussion with the ICANN executive with the ICANN Board of Director during the last ICANN in Cancun. And they are very, very aware, the ICANN, about that.

And last thing perhaps, which is important, so there is still the AFRINIC staff. They are making a great job, really. And, one, so it’s important to organize election, as you said, but it is very important also to support so the AFRINIC staff because it’s more than important that they can fulfill their functions. Thank you.

John Curran: Thank you. Agree. Thank you. Very good.

Front and center. Mike?

Mike Burns: Mike Burns, IPTrading. A couple of thoughts. One, you said AFRINIC is doing a great job from their inception, but their CEO was convicted of fraud at some point.

John Curran: I’m sorry?

Mike Burns: Their former CEO, not Eddy

John Curran: Neither their CEO nor their former CEO were convicted fraud. There was a staff member who was let go on allegations of fraud. And that’s a gentleman several years back who was working in the engineering organization. Not the CEO.

Mike Burns: Some of the claims against them relate to some of his activities regarding, you know, legacy addresses. So there were some mistakes.

John Curran: And they changed, they brought in a new CEO, and they had an in-depth registry audit to make sure they knew the extent and scope of whatever went on.

Mike Burns: Yeah. I have two other questions. One, the issue they apparently are having with Cloud Innovations relates to something in their contract relating to continued use of addresses for their originally intended purposes and not for abuse in the original allocation to Cloud Innovations.

And the language that requires IP address holders to continue to use their addresses for their original intended purposes is problematic for, you know, for many people.

But I wanted to mention the money. When the AFRINIC bank account was frozen, it was — I’m not sure if it was the ASO or NRO; the other RIRs have an emergency fund ?

John Curran: Yes, we do.

Mike Burns: Did they pay anything to AFRINIC?

John Curran: So the challenge that you have is that we actually did tell AFRINIC: If you cannot pay accounts, we can arrange accounts. And in fact, both the RIRs did that, and so did ICANN, stepped up to say we actually made provisions for the emergency pay provision of payroll, if necessary.

We did not trigger the RIR Stability Fund formally. And the reason was because it’s a situation where the RIR Stability Fund was designed for if an RIR knows it needs a loan or if the RIR decides it needs some technical support, it asks for it and we lend it to it.

When you have a frozen bank account, you’re in an interesting situation. You receive a loan, it goes into a frozen bank account, it stays in a frozen bank account. We had a number of things that were problematic with the provision.

Even now, when AFRINIC got to the beginning of the year, there was a question about whether it could draw its own accounts because it hadn’t had an adopted a budget. So you have a circumstance where you’re drawing money out, but you don’t have a budget for the year.

There was some question there. We said, look, we’re willing to backstop that. We couldn’t use the Stability Fund for that either because that requires a resolution of the Board, and there’s no Board.

So we’ve been offering financial support. It hasn’t been through the RIR Stability Fund vehicle. The RIR Stability Fund vehicle was perfectly suited for what it was. But the scope of problems, like lack of a governance issue, is beyond what was envisioned for that program.

Mike Burns: So how much money of ARIN’s members have been sent or spent in AFRINIC?

John Curran: I’m going to ask this: Spent directly on AFRINIC expenses or spent in support of AFRINIC in general?

So, for example, ARIN has provided, we have retained legal counsel in Mauritius, which provides a second view at documents that AFRINIC has asked us to provide on occasion.

We have more than $250,000 in the last year that we spent on legal and associated support.

Mike Burns: Okay. And that’s not a loan not expected to be paid back.

John Curran: That’s spent.

Mike Burns: Okay. And finally, in a worst-case scenario, where Larus wins and is granted compensation greater than the amount in AFRINIC’s bank account, what is the worst possible outcome here?

John Curran: That’s a great question. Let me try to answer it as succinctly and politely as I can.

The stability of the RIR system will be maintained no matter what. So if there’s something that prevents AFRINIC from operating as an RIR, the other RIRs will take action to make sure that the entire set of registrants for the registry system, the Internet number registry system, will still receive service. How that occurs depends on the exact nature of what the worst case is.

Mike Burns: But there are provisions in ICP or whatever to handle this case?

John Curran: The ICP 2, the document that allows for the recognition of the RIR, envisions recognizing an RIR and an RIR then successfully operating on an ongoing basis under its own governance mechanisms.

ICP 2 does not explicitly cover what happens when an RIR that makes representations in ICP 2 no longer fulfills them. So we’re in a gray area there.

But an RIR does make a long list of representations and is expected that those representations aren’t seven-day representations. Those are for the length of its operations.

Mike Burns: Could the RIR be owned by a private individual?

John Curran: One of the representations is a membership-based entity. So that’s pretty unlikely.

Mike Burns: Thanks.

John Curran: Any other questions?

Hollis Kara: We do have a couple in the virtual queue.

John Curran: Sure.

Beverly Hicks: Sylvain Baya from cmNOG: What are the next steps in how the INRS and global community could help more?

John Curran: So there is a motion right now that’s been filed, as I said, by one of the directors of AFRINIC who is supported by the RIRs, supported by many organizations, the Council of Elders that supports. That will go a long way towards resolving this because it will call for an election.

If that pleading is granted favorably, AFRINIC would have an election this summer and then would have a second election, its regularly scheduled one in the fall, which would give us a fully seated Board. That should be the right outcome.

Right now it’s a legal matter before the courts. And there’s not much any of us can do other than wait for the outcome of that.

If indeed that’s favorably granted and there’s an election, we need to all make sure people who are on the continent of Africa know that their RIR is having an election and to get involved, learn, and get engaged so that they can make sure that they’re participating in the election. We want a widely supported, open, transparent election process. Thank you.

Hollis Kara: One more question for you I think.

Beverly Hicks: From virtual, I have Donnie Lewis from The Obsidian Group: Which courts are able to create legal rulings concerning AFRINIC?

John Curran: I’m sorry, repeat.

Hollis Kara: Which courts are able to create legal rulings concerning AFRINIC? So what court is adjudicating

John Curran: Right now it’s before the Mauritius court. AFRINIC is located in Mauritius, and so it’s the Mauritius Supreme Court.

Thank you very much.

(Applause.)

Hollis Kara: Can take a brief pause here.Apologies.

John Curran: Thank you very much. I just wanted to make sure that everyone had their questions answered, and we were doublechecking on it. Have a good day. We’ll now move on to the next presentation.

(Applause.)

Hollis Kara: I’m going to click ahead here. Next up, real quick, as Brian is coming up, earlier this year, Brian and some of his colleagues at Virginia Tech reached out to us and said, ‘Hey, you think you could help us put together some promotional materials? We’ve reached a really big milestone, and we’re really excited about it.’

And we went, ‘Wow, that’s really cool, we’re working on an agenda for ARIN 51, and we’re really excited about your milestone too.’

So we’ve invited them here to talk a little bit about what it looks like and feels like to be one of the few people that can brag that they’ve got 25 years of production IPv6.

IPv6 Success Stories

Keynote from Virginia Tech

Brian Jones: Hello. I’m Brian Jones, Virginia Tech, ARIN AC, here to talk about Virginia Tech’s favorite network subject, IPv6.

Don’t want to take any more time. I’ll go ahead and introduce you to Dr. Scott Midkiff, who is going to tell us a little bit about his part in that IPv6 history.

Dr. Scott Midkiff: I especially want to thank ARIN for giving Virginia Tech this opportunity to discuss running IPv6 in production for 25 years.

We all know that the transition to IPv6 has been lengthy, to say the least. The good news is that IPv6 is much more pervasive today than it was 25 years ago when we were definitely an outlier. But the bad news is that IPv6 is not yet universal. We still have vendors and Software as a Service providers that do not fully support IPv6.

Even though I did not become Virginia Tech’s CIO until 2012, the 25th anniversary of IPv6 is still special to me. I was a faculty member in Electrical and Computer Engineering on the team that led to Virginia Tech’s production deployment of IPv6 in 1998.

Two of our ECE Ph.D. students, David Lee and Dan Lough, worked with network engineers, including Phil Benchoff and Eric Brown from the university’s IT organization, to explore, test and deploy IPv6 at Virginia Tech. We’ll hear more from Eric and Phil after my remarks.

My predecessor as CIO, Erv Blythe, recognized that IT had an important opportunity to advance innovation and research competitiveness within the university.

IT and an academic unit, in this case the Bradley Department of Electrical and Computer Engineering, worked together to move an emergency technology, but not a new technology, from a somewhat immature state into practice.

It allowed Virginia Tech to skate to where the puck is going to be, to use the often-quoted phrase from Wayne Gretzky.

Virginia Tech continues to be committed to innovation and research competitiveness. We support researchers advancing cybersecurity through our IT Security Lab. We’re working with the Commonwealth Cyber Initiative to deploy an outdoor wireless testbed to explore production uses of CBRS spectrum and private 5G and to enable Virginia researchers to advance next-generation wireless networks.

And we’re certainly strongly committed to IPv6. We continue to work with our vendors and other institutions to make universal deployment a reality.

So thank you again to the program organizers for including this segment in the ARIN meeting. I’ll now turn things back over to Brian Jones.

Brian Jones: Thank you, Scott. I think our next presenter is Phil Benchoff, is that correct? I believe so. Phil, aka “The Beard,” most of the good technical discussions that we have these days happen on his back porch. He’s a great innovator, a great mentor, and a great friend of mine.

Phil, do we have you online?

Phil Benchoff: I am here. Can you hear me?

Brian Jones: Yes.

Hollis Kara: We can.

Phil Benchoff: Excellent. I’m Phil Benchoff. I’m a senior escalation engineer with Virginia Tech Network Infrastructure & Services. I started working with IPv6 in 1997 or so, and I really appreciate the opportunity to share some of what I’ve learned during that time today.

I plan to focus on just a few topics that I think are important today rather than a lot of details about our history. Slide, please.

As background, we have definitely put time and effort into IPv6. But it was nowhere close to the hardest and most labor-intensive things we have done. Paying attention to incremental deployment and risk management has minimized the costs and spread them out over a long time.

Many of the really difficult-to-deploy technologies that we have worked on are long gone. IPv6 is still here, and it’s still important.

For the average organization today, making progress with IPv6 is really more about asking vendors the right questions and testing things than it is overcoming obscure technical issues.

So there are some things you can do today no matter how far along your IPv6 deployment is. Probably the most important one is just ask about IPv6 in your procurement process.

Sooner or later you’re going to start using IPv6. And everything you buy today that does not support IPv6 is eventually going to have to be dealt with in one way or the other.

The top vendor excuse today is still “Nobody asked about it.” We know that’s not always true, but it certainly is in some cases. Zoom, by the way, is now available over IPv6. That is a fairly recent development. And I think that’s at least partly because we asked them about it a few months back.

If your vendor doesn’t support IPv6, ask them for a roadmap. And when you look at a vendor that doesn’t have IPv6, consider what else they might have been ignoring when they were ignoring IPv6.

The way you think and talk about IPv6 helps make sure expectations are clear to vendors and others within your organization.

It’s about where we really want to be with the network and not what we have to put up with in the interim. A phrase I got from DREN is that they want to be an IPv6 network, and they only use legacy IP to temporarily fill gaps. I always try and take that view myself.

IPv6 should be viewed not as an option or an add-on, it should be there by default.

Legacy IP is a liability. You have the cost of addresses. You have the complexity introduced into networks by things like NAT and RFC 1918 addresses that are not present in IPv6.

This is my first ARIN meeting, and I’ve been amazed at how much time here has been spent dealing with issues related to the scarcity of legacy IP addresses, including the part about AS112, which, if you really think about it, is all about the fact that we’re reusing those addresses in other places.

Finally, IPv6 really works today. You can look at many large mobile carriers, ISPs and service providers and see that they have IPv6 running everywhere. So you can’t really call it experimental or untested.

No amount of asking those questions produces results if someone doesn’t also test. A claim of IPv6 might mean anything in between answering ping and trace routes all the way up to full support.

It is fairly common that what you find is that the main features have been tested with IPv6, and they’ve probably been tested on a dual-stack network.

So the most significant test is see if things work on an IPv6-only network. This way you’ll also know that management, updates, infrastructure, and support services, like DNS, NTP, authentication, and licensed servers are also available over IPv6.

If a product isn’t quite ready for IPv6, try and verify that it works with NAT64 and DNS64. And I think you should question a vendor’s commitment to IPv6 if their primary website does not support IPv6. This has actually been a pretty reliable indicator for more than a decade.

So along the lines of how you can get started without breaking everything, the most important thing is IPv6 isn’t some global switch on your network. You generally start a deployment at your network border and incrementally expand that into the organization.

We found that seeing IPv6 work on real networks is probably the best way to build confidence for larger deployment. Once you have IPv6 working on a few services or a few desktops of the right people, internal resistance can often be overcome.

So to get started, you’re of course going to need to get addresses. And that’s easy enough to do these days. Core network routing we have found to be fairly easy and low risk. Once you get the first few routers talking, everything will fall into place. And because the IPv6 configuration is generally independent of the legacy IP routing configuration, it poses really very little risk to your legacy IP traffic.

As you bring it into the network, we always suggest that the engineering and user support networks get it first. If you have success there, you’ll build confidence in those people, and it helps enable yourself to increase deployment.

For service, DNS is what really counts to enable it with IPv6. You do have some background work to do to make sure that the service has basic IPv6 connectivity.

We would typically introduce a name in an IPv6 subdomain with just a quad A record and use that strictly for testing. Keep in mind, if you are using TLS, you might need to add a Subject Alternative Name to your certificate if you care about that in testing. Turns out that tends to be a lot easier with ACME or other automated provision systems.

When it’s time to go production, you add the quad A record to whatever the primary name is. You can shorten the TTL in DNS, if you wish, and you have a pretty safe deployment.

Long term we do generally maintain IPv4 and IPv6-only names just for testing. A lot of times it’s easier than finding whatever option for whatever command you’re trying to use when you want to know which protocol you’re using to connect.

Interfaces also have kind of some options for you to roll things out incrementally. One of them is until you’re sending router advertisements on an interface, people are not using your IPv6.

Router advertisements, when you turn them on a particular interface, have some options to control the lifetime for the prefix and other things. So you have some additional protection there.

Our experience has been that I don’t believe we’ve ever had any problems, at least in more modern times, enabling IPv6 on a user network. Things just work, provided your backbone IPv6 connectivity is working.

One other technology to be aware of, and that’s called Happy Eyeballs. It basically attempts legacy IP and IPv6 connections at the same time. And if either works, things are okay. That avoids the wait for a timeout of one protocol before falling back to the other.

Happy Eyeballs has been baked into most browsers and a few other things for more than a decade. That was one of the key technologies that helped enable the IPv6 World Launch.

And that is all I have for today. I’ll turn it back to Brian. Thank you all for your time.

Brian Jones: Thank you, Phil. Next up we have Eric Brown. He’s one of our senior network architects. And he’s been around for a while, and he has some good things to say about IPv6.

Eric Brown: Good morning, ARIN 51. My name is Eric Brown. Thank you for allowing me to contribute virtually.

I’m a member of the engineering team that operates Virginia Tech’s IPv6 and legacy network. Today I want you to help make adopting IPv6 easier for everyone else. First slide, please.

No talk about IPv6 can be complete without a call for adoption. My first bullet point is a call to urgency. I can imagine two different futures.

In one, there’s no flag day. Progressively, more and more organizations decide to turn off legacy IP as they feel ready and the market is ready for them. As more and more do this, we will find our reliance on legacy IP dwindles, and eventually we simply turn off legacy global routing. We achieve a soft landing.

In the other scenario, we keep pushing forward, avoiding change. In fact, we spend an awful lot of effort avoiding change. We’ll probably have whole engineering teams set up to manage ever-increasing complexity associated with managing and shuffling around ever-dwindling address resources.

Well there is only so much more rope we can tie onto the end while we’re racing toward it. I don’t see this ending well. I see panicked change to IPv6 at the end as large networks outages become an all-too-common experience.

In reality, I doubt we’ll see either of these extremes I imagined, but the engineer in me would most certainly like to see something more like the soft landing I imagined.

Instead of the usual way of pleading for users and operators to adopt IPv6 and reiterating all the tired arguments, I’d like to turn it around and look to see what things could be preventing or delaying others from adopting, where are the barriers, what’s causing the friction.

In this part of the keynote, I’m going to examine different parts and aspects of our industry and call out some things that I think are causing others to pause when it comes to adopting IPv6.

I’m not trying to point fingers here and tell anyone what they should be doing. I certainly don’t understand the complexities of each of these areas.

Rather, this is a call for self-examination. We should be asking ourselves what can we do to make IPv6 adoption easier. Next slide, please.

I’ve included Greg Ferro’s tweet from a couple weeks ago because I thought the irony was funny. In my generation, I think this qualifies as “ha ha” but true.

The reality is addresses should be a common good with the regional registries as the responsible administrators. Clearly, when we’re facing exhaustion, this cannot be so. Don’t get me wrong, I think the market for address space is necessary. It’s one more way to wring some more time out of dwindling address space by creating incentive to free up unused space.

The problem with markets, though, is that they take on a life of their own. They create incentive structures that resist change. In this case, the change is to adopting IPv6. Unlike most markets, this is a market that was never intended to exist indefinitely.

I’ve labeled this friction “perverse economics,” a market that was created to aid the transition in technology now has the potential to resist that very change. Next slide, please. I couldn’t resist including a picture of the cake.

For the second area of friction, I’m going to look at the large transit providers. There’s a basic promise in the Internet that when I hand off a packet to my carrier, they’ll find a way to deliver it to its destination. There’s a basic trust that this will happen.

With IPv6, users aren’t so sure that their traffic will be delivered. We see this when we compare routing tables from various carriers. Reachability is not the same.

I don’t operate a large transit network so I don’t understand all the goings-on. It is apparently about contractual disagreements. I do, however, know that the result, this results in an optics problem.

Users don’t think IPv6 is ready because these reachability problems create a distrust that my traffic that is important to me will be delivered. I did a quick search and found several articles readily and discussions covering this. I’m sure you can find more.

Let’s find a way to remove this friction. Let’s stop weaponizing reachability. Next slide, please.

Here is another challenge people face when trying to use IPv6. How IPv6 is delivered to end users has a lot of variability. In many cases still, it is simply not available. But even where it is available, there are other problems.

I’ve noted two here. One, nonuniform availability, such as a carrier providing it for residential but not business tiers.

And a second more subtle problem: In many use cases, prefix delegation is a must, but an unstable prefix just creates headaches.

I am sure there are other problems. ISPs should take a long, hard look and see if they are causing any. Next slide, please.

In my opinion, this may be the single most impactful change right now to bring us to the next [phase of IPv6 adoption.] Application and content providers should be on the leading edge of enabling their services for IPv6.

This is important because those of us that provide general network service to users and consumers of their content can start to consider going single stack.

Don’t worry if you think no one is asking for it. Your end users don’t actually know or care whether they’re connected over IPv6. I’m not just postulating. I’ve seen this in practice for years.

IPv6 should be a first-class citizen within the application and content provider. This starts with management commitment making it de rigueur and advertising it.

It is in this case, baby steps are certainly appreciated. First thing is to make sure your application doesn’t break for NAT64 users. If you’re unsure, the gold standard is to set up a single-stack test users.

In the medium term, this will lead to the freeing up of legacy address resources as end users’ networks start to see dwindling demand. Next slide please. One before that.

We used to joke that we asked for IPv6 “parity” from our network vendors, but they only delivered “parody.” For the most part, IPv6 support has become consistently good. On behalf of many network operators, I say thank you.

But there are still vendors adjacent to the networking space that want to sell goods and services. The situation isn’t quite as good there. We still see silly problems that aren’t being caught in production that purport in products that purport to support IPv6.

The point I’m trying to make is assume IPv6 is business critical for your customers. Use your own products in an IPv6 environment. You’ll catch your own mistakes.

Lastly, I’ll reiterate one of Phil’s points we referred to as Ron’s rule. If your website isn’t accessible by IPv6, then you’re already, then you’ve already started the relationship you’re trying to establish on the wrong foot. You’re showing me that IPv6 isn’t important to you.

Next slide. I probably should have mentioned this friction along with application and content providers. The line between software and online services continues to blur, but software deserves its own mention.

Again, the gold standard is test your software in a single-stack environment across its entire lifecycle. Make sure installation, registration, use, and updating all work correctly. Next slide.

I’ll turn my eyes to end users and operators. They can also help themselves. Always seek out solutions to your problems. There will be hiccups, but others have gone through this and are around to offer support.

Don’t be afraid to learn. You are unlikely to see as many problems as we did. Whatever you do, don’t turn it off as a way to avoid problems.

Don’t be spooked by IPv6, just turn it on. You’ve already implemented a half dozen technologies that are more complex in the time IPv6 has been around. When you talk to your vendors, demand that they support IPv6. They need to hear it from everywhere.

And the last bullet is a note of warning. As availability for legacy addresses gets more complex and harder to get, it will be the small operators and users that suffer the most. Next slide.

Now it’s time to point the finger back at ourselves and education. There’s a lot of education that still teaches networking using legacy IP. Even at the university level, where the goal is to understand to be able to analyze the dynamics of any protocol, familiarity is still a good thing.

The idea that IPv6 looks scary has a very outsized effect on people’s decision making process. Once you get to know it, it’s not scary at all. And finally, last slide, please.

At the end of the day, the biggest hurdle left to full IPv6 adoption is just the will to do it. We can do this. Yes, you will experience technical issues. Most of the issues have been experienced before. Seek out the answers.

There may be some new issues yet to uncover. Don’t lose heart. None of them are unsolvable. IPv6 is a proven technology, and it’s ready for business.

I apologize if any of the things I’ve said have stunned or come off as provocative. My heart is to do the right thing, and I meant everything here constructively. I’m looking forward to a soft landing. Thank you very much.

(Applause.)

Brian Jones: Thank you, Eric.

Dr. Midkiff, did you have some closing remarks you wanted to make?

Dr. Scott Midkiff: Just again, thanks to you, Brian, for representing Virginia Tech at ARIN and for facilitating this presentation. Thank you to Eric and Phil for their comments, their inputs, and especially their technical contributions over the past 25-plus years.

And thanks again to ARIN for having us, and we are looking forward to seeing greater adoption, full support for IPv6. Thank you.

Hollis Kara: Thank you.

(Applause.)

Congratulations on 25 years of production IPv6. That’s amazing, guys.

Absolutely, Richard, come on up.

Richard Jimmerson: My name’s Richard Jimmerson, I’m with the American Registry for Internet Numbers. I don’t come to the mic very often, but about 10 to 15 years ago this community put the ARIN organization into a very aggressive, heavy IPv6 outreach effort. Did a lot of outreach, went to a lot of shows, talked to a lot of people.

And I just wanted to say thank you to the Virginia Tech organization because in all that work that we did on this community’s behalf, what people are looking for and what they wanted to grasp onto was proof that there was somebody out there that actually deployed IPv6.

They had a proof-positive demonstration to folks that, yes, it’s working. Yes, you can go. And I’d like to say that we crushed on Virginia Tech quite a bit, and it gave many companies the confidence that they needed to do their very own first IPv6 deployments.

And I’d like another great round of applause to the Virginia Tech organization and the work they’ve done.

(Applause.)

Hollis Kara: Thank you again. Thank you guys for joining us virtually.

All right. So we’re coming up on lunch break a little bit late. We’re still going to reconvene at 1:30.

There are a couple of things I want to let folks know so nobody’s surprised. You may have noticed earlier we had a minor blip with our network. We’re going to be doing a little bit of work in the second half of the lunch block, between 1:00 and 1:30.

So for those who are on the Zoom, you may need to reconnect. We are still going to be having our virtual table topics, so those are still going forward as planned.

If you’re here in the hotel, you might want to switch to the Hyatt Wi Fi over the lunch break for a little bit so you don’t have any service interruption if you’re doing any work for back at the office.

So with that, lunch is outside. And we will reconvene at 1:30 for our IPv6 Success Stories panel.

(Lunch break from 12:12 PM to 1:30 PM.)

Hollis Kara: We’re going to go ahead and get started. We are back from lunch. Next up we have a panel presentation, which is going to be great. It’s all IPv6 success stories, growing with IPv6, and John’s going to come up to moderate and introduce all of our panelists.

IPv6 Panel: Growing with IPv6

John Curran: Good afternoon. Welcome back from lunch. We’re going to have a panel presentation. I’m looking for Matt Wilder. If you’re out there somewhere, you can join us. And Brian, I know if you’re out here somewhere, come on up guys. We have chairs for you and everything — microphones. Look at that. And we have our remote panelists.

I’m John Curran, CEO of ARIN, and I’m going be moderating this. It is the Growing with IPv6 panel. I’m going to introduce the panelists and give them each a chance to have an opening statement. And then I’m going to pepper them with questions until they cry. No, I’m going to ask them to explain a little bit about their IPv6 journey.

With that, let me introduce the panelists. First one is Ben Bittfield. I’ll give him an opportunity to talk about the IPv6 experience. Ben, why don’t you introduce yourself and explain the experience at your organization. We need audio. We’re working. Hold on, Ben. We’re working. One more time. Beautiful.

Ben Bittfield: As stated, I’m Ben Bittfield. I’m the IPv6 engineer at Johnson & Johnson currently. Previously I was at T-Mobile and Sprint. That’s predominantly where my success has been.

I’ll briefly review that and talk about where we’re looking at in the future at Johnson & Johnson.

At Sprint, over the last decade or so, we walked through the various — basically all the different migrations times we’ve had from v4 to v6. And there’s a case study out there at ARIN that I wrote basically on our journey through the different transition types.

We started out with public IPs. Went through the carrier-grade NAT phase, dual-stack and eventually our target state of v6 only. So went from a phase, where, when I first started there, we were on flip phones and we didn’t hardly have to give an IP address to a flip phone. So we could oversubscribe public IPs many, many times over.

But when smartphones came around, we were really in a bind where we had to figure out some way to get off of that. We went the carrier-grade NAT route because it just jumped on us. Smartphone adoption was very aggressive. And so we had to figure out something quickly.

Our target state was always a v6-only world, but it took a while to get there. But we went from a situation where we basically did not control our own destiny.

We had the unfortunate requirement, I shouldn’t necessarily requirement, but we were forced to squat on Department of Defense address space. No one wanted to do it but it was the easiest, fastest way to get it in the network. Once it’s in the network, it’s out there, right? So we went from that, where we had considerable risk, to where, in a v6 -only world, where, on 90 percent of our handsets, we didn’t have to worry about that risk of losing all of our subscriber data.

So I definitely slept better at night after that. We controlled our own destiny again. It was a nice situation to be in.

Along the way, we noticed, and there’s been other studies and panels and discussions on this, but we saw definitely some better latency improvements when we rolled this out. We were also able to fix a lot of — simplify — the network architecture, make it simpler, flatter, cleaned up our IPAM.

Like our Internet-facing ACLs and some places, we had 200-plus either NAT pools or public IP ranges; whereas, with v6, we could summarize it in the single /36 or something. Definitely simplified a whole lot of things across the board.

And a major benefit of that — which is not in the case study that I have written — but the merger between Sprint and T -Mobile, both parties being predominantly on v6-only handsets, was a huge, huge win in tying the networks together.

We had two major telco networks, both with 50 million-plus subscribers. If you try to integrate two v4 networks, with that large of mobile IP pools, that would have been a complete nightmare. We would have been back to having to squat on DOD space or federating the network and reusing the same IP range numerous times over.

But the fact that we were both predominantly on v6, it was definitely — not to oversimplify the work involved — but it was a matter of: I need to route this IP range to this IP range and that’s it.

For the v4 networks, it was NAT upon NAT upon NAT and just layers of nightmare trying to integrate the v4 networks. But for our v6 handset pool, it was just “I need A need to go to B.”

John Curran: You were definitely spared quite a bit there.

Ben Bittfield: Definitely.

John Curran: Go ahead. Any closing?

Ben Bittfield: I was going to say, I’m excited for the future of where we’re going with v6. When I first started at Sprint, we were talking about predominantly tunneling, and dual stack was the new thing, right, not the new thing, but v4 dual stack where you can, tunnel where you must. And now it’s v6-only where you can and dual stack where you must.

I like that v6-only is not just practical, but it’s in production and it’s out there. And it should be the default going forward, if possible.

John Curran: Excellent. Thank you for that.

I’m going to introduce Brian Jones, but we sort of just had the full intro of Brian just a moment earlier. Brian, anything else you want to add before I go to the other panelists?

Brian Jones: Go IPv6.

John Curran: Okay. Go IPv6. Okay. Brian Jones, Virginia Tech.

I’d like to — Oh, you have a little slide here: Is it new? Is it easy? Do you want to go through them ?

Brian Jones: Yeah. IPv6, is it new? No. Is it easy? No. But is it harder than anything else you do? No. Is it worth the effort? Yes. Next slide, please.

John Curran: I can give you the clicker. Green will advance you, red will go back.

Brian Jones: Some of this might be repetitive —

Beverly Hicks: Brian, can you bring your microphone a little closer so everybody on the line can hear you?

Brian Jones: Little bit closer? Is that better?

Beverly Hicks: Perfect, thank you.

Brian Jones: Okay. 1996, R. Hindon and J. Postel work on RFC 1897. This paves the way for folks to get IPv6 test address space to use on the real networks.

So 1997, Virginia Tech, couple of doctoral candidates got interested in IPv6. They collaborated through their department head, which was Dr. Midkiff, which you heard from earlier before lunch, and that’s how it got to the operational side of the house, which is who I work for.

And the collaborative effort there was very important because the operational side that I worked for, we’re a cost recovery organization. So it could have been, hold on, who is going to pay for this. But instead, everybody said, hey, this looks like the future of the Internet; we need to do this.

Eventually it gets to Phil Benchoff, aka The Beard, that you heard before. He says, I think there’s a digital UNIX package that I can download and get this running on my machine.

January 30, 1998, Virginia Tech receives an allocation from RIPE NCC, 5F05:2000, and we began production of IPv6 on the Virginia Tech campus.

For the next 10 years, we work with vendors, build parallel networks, build tunnels, do what we can to support v6 across the campus.

2008, we emerged from the test phase. We get addresses from Mid-Atlantic Crossroads. Virginia Tech, in 2009, became one of the first places where users could reach Google over IPv6.

Our IPv6 traffic really soared during this time. 2010, Google ranked Virginia Tech as one of the top five deployments worldwide based on percentage of IPv6 enabled host and traffic volume.

2011, the university was asked to consult with the United States IPv6 Task Force to offer what we learned about the transition and presented at the 2011 IPv6 World Congress in London.

2012, June 26, some of you may remember, World IPv6 Launch Day. Virginia Tech emerged as one of the top network operators with nearly 60 percent of the Virginia Tech hosts supporting IPv6.

Since then, our deployment has steadily grown. Nearly 90 percent of our network hosts support IPv6 today as compared to 45 percent worldwide.

Thank you.

John Curran: Thank you. Wonderful.

I’m going to introduce our next speaker. That’s Madhura Kale. I know she’s got a conflict coming up right after this, is going to be speaking but may not be on the panel, depending on time constraints.

Go ahead, Madhura.

Madhura Kale: Hi, everyone. It’s really wonderful to be on this panel. My name is Madhura Kale, I’m a Senior Manager of Product at AWS in networking. Been with AWS for just a hair under seven years now. And I own some of our IPv6 space.

John Curran: Very good. You want to go through your slides? I can direct them as you go.

Madhura Kale: That sounds great, John. Thank you. Pardon the simplicity of my deck. I wanted to give everyone a background of what IPv6 on AWS means. I wish I could come and say we launched I will say we launched IPv6 on all of our Amazon retail websites, but AWS is sort of different, where we’re providing the infrastructure that allows different businesses and organizations to build IPv6.

So we started our journey with IPv6 back in 2011 where we have this product called the Elastic Load Balancer, and we said, yes, we launched IPv6 with the rest of the world.

And AWS started in 2006. So five years later, we realized we need v6 support and we launched v6 in 2011.

But moving on. Next, please. We got a VPC. A VPC is your fundamental cloud unit for the construct that you build in, and maximizing it just for sake of what’s coming up.

You have typical applications on AWS for simplicity are built with one. Next, please.

John Curran: Load balancer.

Madhura Kale: Load balancer and the next is compute, which is your server, that’s hosting your application, and last is your data store. Very simple. In cloud world, you typically view load balancer as what’s facing the Internet, where we have several other services, too.

When we built additional IPv6 support with more products — next, please — we thought we’d cover load balancers, DNS, comp and distribution, anything that is facing the Internet, including actually our core compute and storage products. We pat ourselves on the back, saying, “We did IPv6.” This is 2016, “We’re done.” And all of these products supported dual stack.

Now, coming to the next layer, that is the compute, we started seeing our customers scale their very large networks. They became really, really large. Not different from how Ben was describing. Lots of NATing between VPCs.

We’d have customers being extremely creative with how they managed routing space. And then several that said, hey, we want to stop. We want to pull the plug on this. We’d like everything in our internal servers to support, internal applications to support IPv6 and v6 only.

So in the last few years, last couple of years, we’ve launched v6-only for our core compute service. Launched v6-only for our Kubernetes service. And we’ll soon get with v6-only for our serverless compute as well.

And, finally, last but not the least, there’s also customers that say, hey, we can’t just run half of our workload in IPv6 and the rest not, which comes to our data stores. And now we’ve continuously expanded our data store IPv6 support as well.

If we go to the next slide, this is a better representation of the majority of services that support v6. This is still not exhaustive. And this includes hundreds of different features and things you can do with IPv6 on AWS.

But we’re always, always looking to learn more as our customers evolve on what you can do with IPv6.

If you go to the final slide, there’s a QR Code, if folks can see and scan, that’s our IPv6 page that talks about what we do with IPv6 recent launches, customer success stories, as well as it links to a sort of detailed support matrix to demystify the cloud.

John Curran: Thank you very much. That’s been wonderful. Anything else to add? I know you have a time constraint. So I’m going to try to go through people fairly quickly in case there’s questions. But I understand, if you need to drop off during, just drop off.

Madhura Kale: Thanks, John. I appreciate that. I’ll be here for at least another 30 minutes. And if I need to drop, I’ll wave in the camera and go away.

John Curran: Very good. Now I want to introduce Brent Mc Intosh from MCNET-SOLUTIONS. Brett, glad to have you join us. Give a little introduction of yourself and your IPv6 experience.

Brent Mc Intosh: Thank you very much, John and team. It’s a pleasure to be here. Of course, I lived in the Caribbean. I worked the last half of my life with a couple of service providers here, and that’s pretty much where I gained my IPv6 experience.

I’m part of the Caribbean Networks Operator Group. I’ve been doing quite a bit of work on Internet exchanges, with the CaribNOG Group, the Caribbean Telecom Union as well for the last couple of years. And we don’t want to deploy any Internet exchange without having IPv6 on it.

I have been doing quite a bit of work in the last couple of years on enterprise infrastructure, and it’s quite surprising the interest of IPv6 deployment even in the financial sector.

And today, my case study, what I’m going to show you, it’s not an enterprise business, although they’re very popular with requesting IPv6, I’m actually doing sort of a presentation on an Internet service provider because that’s the challenge in the Caribbean which I would want to highlight today. And it’s something that we want to get going: ISPs providing not just transit IPv6 but providing Internet access over IPv6 infrastructure.

So looking forward to sharing with you and learning from you as well.

John Curran: Excellent. Good to hear. I’m going to start in on the questions.

Brent, you have slides, apparently. Do you want to go through them?

Brent Mc Intosh: Yeah, let me go through my slides.

John Curran: Sorry. I was unaware. Here is your “Who is Cable Bahamas Limited.”

Brent Mc Intosh: Okay, great. Cable Bahamas is an Internet service provider in the Bahamas Islands. They had some challenges. And those challenges led to a really good success story.

It’s a 100 percent Bahamian-owned service provider. They provide TV, Internet, and video services. And now major service for access with fiber to the home under the brand called Live Fiber. Next slide, please.

Next slide. Great. So they realized that we’re going to have challenges with IPv4 shortage. And their idea was to say, hey, let’s build an access solution using IPv6. And it started from looking at what was required and building the right team. Building a tech team to get this done was important for them, and they decided our solution was to deliver access to content and Internet services over IPv6.

In the past, they were primarily a CGNAT shop, sort of bandaid for dealing with that IPv4 address issue, but they realize it’s important that CGNAT is only going to be for our transition to get to v6 only at some point.

They’re now at the stage where it’s dual-stack deployment, and that works fine. One of the important things to note is that they did not leave out training for their technology team and also for their customer service team.

And I was really happy to be part of that training process to get their Internet and their customer service reps up to speed because no deployment in IPv6 is going to be successful if your supporting teams and your technical teams are not trained.

Next slide. For Cable Bahamas, they fall under the ARIN RIR. They requested their IPv6 allocations, a /32, and what was neat is they actually broke this up into /36s and they decided we are an ISP and we’re going to be multi-homed. We’ve got 400 gig circuits to our transit providers and so we’re going to allocate our IPs in such a way that we can sort of balance our traffic based on specific BGP metrics.

They built out full IPv6 core, dual-stack full access network, dual stack using FPTX, fiber to the home, fiber to the business. But interesting to note, their legacy access technology, which is DOCSIS 3, they also deployed IPv6 on that as well.

So now their business customers, their retail home subscribers are pretty much now testing IPv6. They have about maybe, I would say, a little less than 10 percent of the Internet access content being accessed over IPv6. And that is only because they have not fully launched the product as yet but that is going to be happening pretty soon. Next slide.

This gives an idea of the IPv6 route propagation. They’re using three main transit providers all doing full IPv6 transit for the IPv6 prefix. Next slide. Right. This just gives a very simple highlight and high-level view of both access and distribution with their BNG layer. Every device that’s in that network is fully dual stack as I mentioned before.

Interesting to note, as well, is that their CDN servers was not spared as well. All CDN servers: Google, Akamai, Facebook, Netflix are fully functional with dual stack serving customers in the FTTX access network and also the DOCSIS network and similarly their business solutions.

DNS for IPv6 is actually, that’s also in progress right now. It’s already up and working, but it’s going to be a full solution that would include DNSSEC as well.

So what’s important about this particular case study and the reason that I selected this one over the other IPv6 deployments is because we need an example of a real, proper IPv6 deployment for service providers in the Caribbean.

While there are many service providers in the Caribbean who can transit IPv6, deliver it as access solution to their subscribers has been a challenge. That’s what this case study really wants to help address. I think the Bahamas will be the cookie cutter of what you’re deployment should look like.

And I would also mention quickly that DHCP v6 or prefix delegation over PPPOE has been of course the choice of delivery to their subscribers. And this works very well. Failover testing. Excellent. As a matter of fact, the failover for communication on that particular network is actually based on BGP EVPNs to deliver the various services for their end users. And this makes our failover really, really a champ for IPv6.

I should have one more slide. I can just wrap up on that one more slide. That was it.

John Curran: Thanks. Thank you, Brent. Excellent introduction. A very interesting deployment.

I’m going tomove to our final one, our final speaker, Matt Wilder, who is right here, who can talk about IPv6 and his experience and then we’ll end up doing some questions.

Go ahead, Matthew.

Matthew Wilder: I’ll take the clicker from you.

John Curran: You have slides, too?

Matthew Wilder: I think so.

John Curran: Oh, I guess so. Yes.

Matthew Wilder: Buckle up guys. It’s a lot of slides. Just kidding.

All right. I broke this into three acts, for those who like theater. Act one, usually you’re introducing the characters and the context. Okay. So I can’t claim to have a 25-year history as my colleague, Brian, and his colleagues presented today.

And actually I wasn’t even working at TELUS in 2003. I cannot take credit for the initial act here which was getting the IPv6 allocation from ARIN initially. That was a colleague of mine who had great foresight. So that’s where our journey began at TELUS.

In 2008, that’s when I entered the IP address management space within TELUS And at that time I was aware IPv4 was running out. We need IPv6, and helped kind of drum up a roundtable around IPv6.

We had technical representation from across different teams and started to talk about what is our path for IPv6 here.

And actually one other piece that happens here is, when you talk about IPv6 in 2008, the question that would come up all the time is: how could you compare this to another problem in the industry or IT in general? I would often bring up it’s like Y2K except that we don’t know when it’s going to happen.

And that was fixed in 2009, when we received warning from ARIN, hey, IANA IPv4 exhaust is imminent. We don’t know exactly when.

Ended up being 2014, if I’m not mistaken. But 2009, it was kind of announced to the community, hey, be aware. This is act one. Act two, we started actually doing some building.

So we got a program struck in 2010. We identified some work streams, so everything from DNS to core network. Starting to think about the edge networks and having that structure to kind of guide and organize our activities.

In 2011, we started training. My colleagues have talked about the teams that enable this. You need your teams to be aware, educated, and familiar with IPv6.

TELUS is a national carrier in Canada, with teams from the West Coast to the East Coast, everywhere in between. Lots of cities. We had lots of in-person training as well as a variety of soft skills, the e-learning portals, to kind of give complementary training options to those who couldn’t get it in real time there.

We also did 6PE, getting the core network IPv6 capable. And then also peering. Peering is something that makes the Internet go ‘round. And you need to have that IPv6 peering to make that traffic go somewhere.

2012 was a big year. Of course, we heard it from a couple of folks that they were involved in World IPv6 Launch. And although we didn’t have the 1 percent of subscribers enabled at that point, we were on our path there.

And we did also have our first enterprise-managed service turnup with dual stack that year. And, in fact, the very first service we did was the sponsorship of ARIN. I think it was ARIN 25 in Vancouver. That was a cool, full-circle moment. So there you go, act two.

Final act, act three: deploying. And this is kind of getting to the mass market. But 2013, we enabled our subsystems and service components. Think DNS as well as some of our IT systems to make sure we can support these networks.

One of our first massive services that we launched was 2014. So voice over LTE, mobile subscribers. Instead of throwing countless private IPv4 addresses at these handsets, we did IPv6 only.

It’s one of those nice cases where you don’t need to worry about accessing everything on the Internet.

You can control kind of those endpoints and v6 only is perfectly fine for that case and it’s a great one.

And finally, 2015 was kind of the year where you actually could see the metrics from the outside in. We launched, in the fall, we launched our residential home Internet services as well as some wireless handsets launched with v6 support.

And just to kind of wrap up here, I’ll point out, our deployment in 2015, the one on the left there, the red line, that shows I think it was from APNIC stats, if I’m not mistaken, or World IPv6 Launch, maybe ISOC, that’s our IPv6 percent traffic from our subscriber base. So 45 percent we reached over that fall.

And at the same time, Google IPv6 stats, you can see it climbed from sub 1 percent to 7, 8 percent, somewhere like that. So really started to bring IPv6 to Canada and haven’t looked back.

So our little tagline at TELUS: Let’s make the future friendly.

John Curran: Thank you very much. A wonderful introduction from all of our panelists.

I want to dive right into the questions. And I don’t know who wants to take it first, but let me offer this one as the first question.

What would you have done differently given what you’ve learned now and what you’ve learned about IPv6, if you could do something differently that would have improved your deployment experience, who has a lessons learned or something they would have done differently?

Anyone? Everyone got it right the first time. That’s awesome. That’s even better.

Matthew Wilder: I’ll just say, the sooner you can get more people involved and aware, the better. You’re going to need that core team.

So I think we generally succeeded, but there’s still some areas where we could have done better. I think procurement would probably be the one that I would point out to where really putting in some clear parameters for your procurement folks to say it has to have IPv6 support. That’s a key one.

So I would say if I were to do one thing differently or go back in time, that would probably be it for TELUS.

John Curran: Let me ask each panelist, I’m going to go around: What was the biggest hurdle to your IPv6 deployment ?

Let me start out with Ben. Ben, what was the biggest hurdle you faced?

Ben Bittfield: I would say kind of a two-part aspect of that, kind of related. In regards to like the IPv4 exhaust concerns, yeah, it was, like you said, like the Y2K with no countdown but also almost crying wolf. I was crying wolf, I guess, internally repeating it. But then we did carrier-grade NAT and that kind of stretched that timeline out.

I would say that was the major issue, is that I kept pushing out the timelines because of the success of carrier-grade NAT, I guess I should say.

John Curran: Okay. Madhura, what’s the biggest hurdle you faced in deploying v6?

Madhura Kale: The biggest hurdle, I think, for me is, as a product owner and networking, when customers come to us and say, hey, we need v6 for something, I have to translate that and say, okay, you know it’s not the network, the network runs with v6. What actually needs to happen is the service that you’re running or the application that you’re running should be able to run using a particular maybe compute type or a different database service type.

So translating from hey, it’s IPv6 at different layers, not that the network is capable. It’s also now we’re saying the applications, the software you build on top, the managed services you build on top need to be v6-enabled, and partnering across the organization within AWS to say, hey, here’s follow all the customers and then take it horizontally somewhere to say, hey, here’s this stack, here’s the part of the stack we need to build with v6. So I think that has been the most challenging.

John Curran: Very helpful.

Brent, what was your biggest hurdle in rolling out v6?

Brent Mc Intosh: That’s a good question. I would say the biggest hurdle for me would have been finding the right approach to convince the exec management team that it should happen.

And it was always a financial decision. And I think the approach was, trying to find the best way to convince management teams that this makes sense. And I don’t think we really got past doing this correctly, but this is no longer a major issue anymore. It’s still an issue, but it’s becoming an easier sell. But that was a major hurdle for me.

John Curran: Got it.

I’ll go, Brian, what was your biggest hurdle? Years of experience. What was the biggest hurdle?

Brian Jones: We echo the AWS sentiments. I wish we had gotten our service folks involved and applications deployment folks involved much earlier. It was a great collaborative effort between the academic research side and the network guys. They were all over this. But how much can you really do over IPv6 until you get your services and applications in place?

John Curran: Got it.

Matthew, what was the biggest hurdle?

Matthew Wilder: I think I’ll quote the famous quote about what you know. So knowledge. There’s the known knowns, the known unknowns, and there’s the unknown unknowns. That’s kind of the area I think that you really have to cover as much ground as quickly as you can on to kind of get the word out to your teams, hey, what are the gaps that we need to close in order to make this a reality for our network, for our customers and go from there.

John Curran: Okay. Let’s talk about convincing executives, and a lot of times the executives ask for something that, well, let’s just say isn’t on the tip of our tongue.

Business case, how easy or hard was it for you to make the business case for v6? And I’ll take it in the same order. So Ben, you get to go to bat first.

Ben Bittfield: Initially it was difficult because how do you put a price behind something, in a business environment where everything is costing us? Luckily, we had the issue of v4 exhaust. And I was able to say, hey, look, 50 percent or whatever the number of our subscribers are at risk of blocking data if we do this.

And you’re going to tell 20 million people that they can’t access the Internet? So that was easy to scare them into, like, hey, this is something we have to do or we risk blocking large amounts of customers.

John Curran: That helps quite a bit when you have to do it.

Madhura, do you want to describe the challenge of making the business case for v6?

Madhura Kale: I would say that I’m still making the business case. It’s an ongoing process. But what has helped overall, in the last almost four years that I’ve been driving this, is really hearing from customers, understanding what they are looking to do.

And we have some very, very motivated customers and businesses that are building with v6 either their on prem data centers or purely on AWS.

So hearing that feedback and really making sure we understand the business impact to our customers, whether it’s lowering NAT cost or it’s simplifying a lot of the complexity they’ve built in their infrastructure. And that very customer-driven approach that our Amazon leadership principles teach us to do has helped me make that business case.

John Curran: Got it.

Brent, I see you nodding up and down. Did you have similar experience with your business case?

Brent Mc Intosh: Oh, yeah, absolutely. And Ben said: I hope you put a price on it. I think that’s the mistake I made. Most of the times I actually put a price to it, right. And up front cost was a lot. So it became again a financial discussion.

Do my clients need this right now? Can my clients work without it? So I started doing comparisons to running CGNAT for long computes, then it started making more sense.

So, yep, while there’s a huge initial cost, in the long term it’s actually much better and it’ll cost you less. We look at costs for training your staff, it really became a financial decision.

But I think the approach is always to prove what it would mean to your network operating efficiently and communicating globally with everyone in the future.

John Curran: Got it.

Brian, dare I ask how hard it was to make a business case?

Brian Jones: Actually, with our director at the time, Earv Blythe, he was onboard, because I think he saw the risk cost being larger and unknown more than, hey, I got a bunch of network guys that I can get right on this.

John Curran: Okay. Matthew, anything for you? Any challenges or lessons learned in the business case?

Matthew Wilder: Business case is such a great question because so much of our decision making as organizations revolves around the ability to say here’s the financial impact of action or inaction.

For us, we were really driven by, and what sort of resonated with our executives, was to say here’s the projected exhaust date for these product lines. When you can start to do that, it gets people’s attention that, hey, we might have to stop selling that service. All of a sudden your product managers sit up and go, sorry, what did you just say?

So really, I would say in our case, and your mileage may vary, but the ability to go and actually look at what the cost of an IP address is, the transfer market, that helps to be able to start paying a dollar value that defines the benefit or cost of, like I say, action or inaction.

John Curran: Right. The cost of inaction is important.

Okay. So you’ve all done v6. You’ve got it out there. You must have gotten someone must have found some surprises. Was there any unexpected upside or benefit that you found; you did v6 and later on you realized you got something you weren’t planning, some win or advantage ?

I’ll start again with Ben. Ben, was there anything you got out of v6 you weren’t expecting?

Ben Bittfield: When we were first rolling it out, there were some studies. I think this was around the same time Facebook and LinkedIn were publishing studies about legacy improvements.

When we rolled out our probes and our testing , we were seeing the same results. We were seeing much improved latency, and just better performance. It was the same number of hops. It was the same peering relationship, but not going through that CGN enabled it to be so much faster and cleaner and better.

John Curran: Awesome. That makes perfect sense.

Madhura, any unexpected benefits of v6?

Madhura Kale: Aside from scale, the fact I could tell a customer that you no longer have to worry about the size of your VPC, definitely a big plus.

I have a story I’d love to share with this audience, with Amazon Retail. When we turned on v6 globally across different countries, people talk about latency.

We actually saw a mixed result in latency. In some countries we saw an improvement. And in some countries we saw an actual reduction in performance or increase in latency.

And our team suspects that it is because where every country is at with their v6 adoption, or we’re trying to get all the v6 requests that we’re getting from these clients are probably still routing through a bunch of NATs to get to us, or tunnels. We don’t know. So we saw mixed results on latency.

John Curran: Okay. Very good to hear.

Brent, any unexpected upside of v6?

Brent Mc Intosh: On the upside, again, it’s primarily just performance and latency from the test clients that we’re working with in that particular Cables Bahamas deployment.

From doing enterprise deployments, started getting tons of security-related questions. And again there are some challenges on that front. But again that’s a different discussion, and maybe that’s downside, but we’ve been finding solutions pretty much to any challenge we got so far. So that’s the good thing.

John Curran: Got it.

I’ll ask Brian, any unexpected upside? I know having us call you and refer people to you about your case study every other week might not be seen as an upside, but was there any other one?

Brian Jones: Actually, that is part of the upside. Virginia Tech got a lot of press because we were early adopters.

I think Phil Benchoff that presented earlier, he’s got a webpage that kind of points out a lot of the press articles that was written. It was really good for the university. It was good exposure. And that was an upside that none of us network guys really thought about.

John Curran: Wonderful.

Matthew, unexpected upside of v6 deployment?

Matthew Wilder: One interesting thing, in particular, it’s that stage of dual stage where you have IPv4 and IPv6 running in parallel. It’s in a way like having multi-homing. You’ve got two address families that your end customer networks can reach out, two address family paths. It’s like having that redundancy that adds reliability, robustness. And our customers like that.

John Curran: Got it. Very good.

I want to dive into some specific questions about technology. So I’m going to use, I guess, a four-letter word, carrier-grade NAT. C-G. Fiveletter word maybe. Some of us, in our rollout, used carrier-grade NAT.

Like Ben, for example. Ben, how did that work for you? Are you a fan, an advocate of carrier-grade NAT? Not a fan? What’s your thoughts on the technology?

Ben Bittfield: Necessary evil. It got the job done. It got us a stepping stone to, it brought attention to the problem because we had been talking about v6, but it didn’t it was, yeah, like, okay, do I need it next quarter? No. Then come back in a year. Back to the business case thing, right?

No, I mean CGN works well, but it was expensive, because you have very expensive machines with lots of memory trying to keep state of all of these sessions and everything. And there was performance issues. Like I said, a necessary evil. It got the job done, but it wasn’t what we wanted as our target.

John Curran: Necessary evil. Got it.

Anyone else want to sing the praises or curse carrier-grade NAT? Madhura, I see you smiling.

Madhura Kale: I actually don’t , to be honest, I don’t have any experience with carrier-grade NAT. Our customers do use NAT on AWS. We launched this functionality for private NAT. And depending on the persona I talk to, the customer, if I’m talking to the infrastructure network admin, they don’t like using that because it makes their job harder and more expensive, versus people that own applications and services.

They’re like, yeah, it works, right? Yeah, it works. As long as it works. And again I see the diverse perspectives there.

But I’m curious almost I’m turning it into a discussion but with Starlink using CGNAT and new services, new ISPs coming up, is it a terrible idea to be using CGNAT ?

John Curran: Got it.

Brent, any CGN?

Brent Mc Intosh: Lots of CGN. I’ve never been a fan of NAT, because since 2012 I’ve been an IPv6 evangelist, and so why am I talking about NAT? And like Ben mentioned, it’s a necessary evil.

Initially in the deployments we saw issues where you had what we call port block allocation and customers at home using port forwarding for the cameras, they would have had issues. But, no, that’s pretty much sort of delicate. But that was the initial issue.

First, again, had I done my approach to convincing management why dual stack would make more sense now, even though you may include CGNAT, the cards would have been different, and that’s how I would have drived my business case. But, again, NAT, not a fan of it, and we need to move to IPv6. That’s all I’ll say about that.

John Curran: Brian, you got any carrier-grade NAT in the process of VT somewhere?

Brian Jones: A little bit. It’s expensive, and it adds a lot of overhead. It makes your troubleshooting harder. I’ll stop.

John Curran: Everyone loves this.

Matthew, carrier-grade NAT?

Matthew Wilder: I can tell Brian is not a fan. Carrier-grade NAT, it serves that purpose — of to me it’s a lever to get off the “I need public v4 train.” As long as there’s a predominantly v4 -only set of end destinations that your customers might need to reach, you need an IPv4 option. And those would to me, those would be you’re going to have to buy public IPv4 addresses or deploy CGNAT. So it’s a matter of comparing those options and considering what works for your customer base.

And there are costs. There are costs that go beyond the infrastructure itself in terms of logging and making sure you can support those customers. So it’s not just a matter of the infrastructure. There’s some complexity to manage with that. But it’s there for a reason. It serves that purpose.

John Curran: Got it. I want to move on. People have touched on performance. I’m going to skip that because it’s very subjective depending on what you’re using and where you are.

Let’s talk about resiliency. Has rolling out IPv6 made your network more resilient? Hopefully so. If not, why not?

I’ll keep the order. So, Ben, you’re up first. Did IPv6 make it more resilient? The same? Or less?

Ben Bittfield: When we were rolling out dual stack, it definitely did because obviously you have two networks. Like Matt mentioned, basically multi-homing, right?

And that was at a time where we were going through a major rip and replace of the network. So there were customer impacts across the board. We were replacing network hardware and vendors and everything. Dual stack came at a perfect time for us because there were some instances where we lost v4 transport or where we lost v6 transport.

And for 70 percent of the end traffic, that didn’t change anything, right? I could still reach Google and Netflix over v4 or v6. So, for us, it came at a perfect time and it definitely added resilience and it helped v6 win favor with executives for us to get to our target state of v6 only.

John Curran: Madhura, do you want to comment on, did v6 make you more resilient or the same?

Madhura Kale: From an AWS perspective, pretty much the same, I think. We have dashboards in our monitoring. Anytime I’ve looked at v4, v6, pretty consistent over time. We might have an occasional outage that’s v4-only or sometimes v6-only, but overall I think, in percentages, I think they’re equally performaning.

John Curran: Brent, your experience?

Brent Mc Intosh: So the way we deploy v4, IPv6 pretty much followed the same path. We dual stack everything. So not much difference in the resilience, I would say. But what I would add, though, where we deployed Internet exchange points, there are no services on an Internet exchange point that we deployed that didn’t have everything dual stack. And, of course, most of the applications you access, including CDNs, always tend to prefer IPv6 first.

And there’s the perceived improved performance, which I have seen from testing and then that is an added benefit in terms of resilience.

Well, if my service provider goes down, I can still get locally cached content over IPv6 with no NATs involved. So better performance.

John Curran: Brian, more resilient with v6 on?

Brian Jones: Yeah, I think it did add some resiliency for us. And it helped us convince some of our service folks that, hey, v6 was working while the v4 was down. So I think, yeah, it has helped us.

John Curran: Matthew?

Matthew Wilder: I would say definitely, as mentioned, it was an unexpected benefit for us to see that kind of resiliency that comes from having those two address families put to use at the same time.

Not that we experienced it, but imagine if there was some kind of BGP hijacking event. It’s only going to affect one address family probably because it’s a rare occurrence. So again it’s that redundancy that we all like to see.

John Curran: Got it. Good to know.

I want to ask a couple of targeted questions. Brent, is there anything in the Caribbean that makes deploying v6 harder or easier?

You’ve got a unique environment there to some extent, and you might have some unique factors we want to hear about.

Brent Mc Intosh: What I’m about to say, I’m sure it’s a common thing, but I can tell you it’s really hampering the Caribbean’s deployment.

I’ll tell you what, there is no challenge with acquiring ARIN IPv6 resources. Companies in the Caribbean are actually doing that right now. Everyone is grabbing their resources, including service providers.

The challenge is the deployments. And I can tell you from what I’ve seen in the last three years, I’ve been getting more requests for support for deployments from enterprise clients.

The challenge is not all service providers are ready. And that’s why I chose that use case of Cable Bahamas, because they’re ready. Now the other players need to be ready. While they can transit IPv6, they can’t deliver it as an access solution. And that’s the problem.

So I think, in the Caribbean, we need to pick up the pace from the perspective of our ISPs to be leaders in IPv6 connectivity. And that’s where the challenge is.

John Curran: Perception issue. Very important.

I’m going to turn the same question in some extent to Brian. Universities and colleges have faculty and dorms and students and researchers and all sorts of interesting colorful things that you don’t find in commercial networks. If you were giving the advice to another university or college, is there any particular department or any particular set of educational institutions you’d have them pay exceptional attention to?

Brian Jones: As long as the dorms can get their Xboxes connected they should be good.

(Laughter.)

That’s kind of funny, but it is one of those things where some of those devices that are tough to autodeploy can be challenging at times. So you may have to have you may have to develop some ways to do those things.

John Curran: Got it. Okay.

I’m going to ask one final general question of all the panelists, and then I’m actually going to take a few questions from the floor if people have them.

My last question is: What one piece of advice would you share with your best friend or coworker about if he was embarking on a v6 project with bringing into his new company, what’s the one key thing you’d tell him to pay attention to?

So Ben, pay attention to what?

Ben Bittfield: Man, putting me on the spot here. I would say, in this day and age, security.

John Curran: Security.

Ben Bittfield: Just because security is just a whole different breed of cat in a v6 environment compared to the v4 world. So definitely get your security teams involved early on.

John Curran: Yep, got it.

Madhura, what would you if you’re giving advice to a coworker or someone going to a new company and about to embark on v6, what key piece would you tell them to pay attention to?

Madhura Kale: I would say similar to what Matt said. Talk to a lot of people, involve more people in the company, and don’t take it as your singular crusade to take on. Yeah.

John Curran: Got it. Brent, what’s the key bit of advice you’d give as he’s walking out the door?

Brent Mc Intosh: If that person is not the CEO of the company, I would tell him make sure you’ve got a compelling business case to prove why it would make sense doing it and how it will keep you competitive as a company.

John Curran: Okay. Good to hear.

Brian, what’s the key bit of advice you would share with a coworker?

Brian Jones: Go through the ARIN training on the ARIN pages and think about how you are going to impose the IPv6 hierarchy on your network. Think about your addressing plan.

John Curran: Addressing plan. Very good.

And Matthew.

Matthew Wilder: I’ll just reiterate that having a team of professionals and technical folks and product folks kind of gathering to understand the problems from various perspectives, how that deployment will take shape. The more folks you can get involved, the earlier the better.

If you want to go fast, go alone. If you want to go far, go together.

John Curran: I’m going to open it up to questions from the floor and remote. Hopefully we have a way of doing that. Maybe not. If you have questions, come on up. I see one coming over here in the kleig lights. I think I’ve got you emerging.

Ralph Bischof: Ralph Bischof, Leidos Corporation, NASA contractor, e.root. I’m interested in two things, really. Number one, your actual IPv6 implementation plans. How hard was it for you? How long did it take to do implementation plans?

And I have a second question, if you don’t mind. SLAC, was that something that everybody was well to do, or did you try to do some other protocol for your address assignments?

John Curran: First question is: your implementation plan, how long was it, how long did it take?

Ralph Bischof: Sure. How hard was it to actually and how long.

John Curran: Second question is SLAC?

Ralph Bischof: Is it something that everybody decided to do as far as address appropriation, or did you try some other things as well?

John Curran: Got it. Address allocation and SLAC.

Who wants to pick that one up first. How long was your IPv6 the implementation plan, how hard was it to do?

Matthew Wilder: I guess I can take a stab at it. Doing the math in my head, I think it was roughly five years from the time we got going on it. So it took some time.

And as far as address assignment to our end users, it really depended upon the product and service. So we do have managed Internet service for enterprise. In that case we give a static, of course.

We used DHCP prefix delegation for our residential Internet customers and a variety of others in other cases.

John Curran: Someone else want to pick up how big their plan was and did they use SLAC. Brian?

Brian Jones: We used SLAC. Still use SLAC. Our implementation plan, it’s 25 years plus. Still going. And we were incremental. It started out between an operations group and an academic team. And we just found it was easier to add on incrementally as we went.

John Curran: I’ll turn it to Ben. Ben, how long was your implementation plan and for end device assignment, did you use SLAC?

Ben Bittfield: Implementation, I want to say three or four years, just same thing trying to plan it out as best we could. And we did a slow roll on consumer devices and uncovered some major vendor issues. We had to roll everything back and that delayed us a year or so waiting for bug fixes. So just test, test, test, test, test.

But for us in the mobile space, 3GPP standards basically defined that you use SLAC. It’s a different environment. But SLAC worked very well for us. So I’m a big fan.

John Curran: I was going to say you have a different kind, not a lot of end devices in the same idea of a desktop.

In terms of Brent, how long was your implementation plan, and did you use SLAC for end device assignment?

Brent Mc Intosh: It varied per client. Service providers actually can take anywhere from two years to three years, working side by side.

Enterprise customers, anywhere from one month to six months.

In terms of address assignments. For my use cases, it’s primarily DHCPv6, stateless, stable and some static, believe it or not, because I do have small clients as well that use static assignments.

SLAC is pretty much we use it in the lab and for training but not really in deployments.

John Curran: I see enough people lined up. I’m going to close the queues. But let me move on. Does that get your question?

Ralph Bischof: Yes. Thanks for your time, expertise, and replies.

John Curran: Center mic.

Roman Tatarnikov: Roman Tatarnikov, ARIN Fellow, and in consulting, mostly dealing with wireless ISPs.

I heard you mention DNS64 and NAT64. My question was, how did it exactly work out for you? What cornerstones have you noticed? And what do you think about XLAT 464 and have you implemented it?

John Curran: Okay. So it’s the DNS battle over v6. We’re going to talk about DNS64. We’re going to talk about XLAT. Who wants to go first? It’s very quiet out there.

Matthew Wilder: I can speak to. in the case of TELUS, we’ve done all of it, actually. We’ve tried both methods. So DNS64, NAT64 is kind of an interesting way to synthesize quad-A record replies, basically. So you synthesize it and you correspond it and it matches up nicely.

We ended up sort of sunsetting that and going instead toward 464XLAT, because it covers all the cases you might want, including v4 literals. That’s the key that 464XLAT solved. We ended up sort of winding down the DNS64/NAT64 piece in favor of 464XLAT.

John Curran: Brian.

Brian Jones: I need you to talk to Phil Benchoff, our DNS expert.

John Curran: Not a problem. That’s an answer too.

Brent?

Brent Mc Intosh: This one is easy for me. I haven’t looked at any mobile deployments as yet. XLAT would definitely be an option for that. And I’m thinking it’s going to happen maybe in the next couple of years with this particular client I’m working with.

DNS64, NAT64, small implementations, but we tend to stick dual stack everything including DNS servers.

John Curran: Kind of a specialty. Hope that answers your question.

Roman Tatarnikov: It does, thank you.

John Curran: Okay. I’m going to go to the left microphone.

Lou DeVictoria: Lou DeVictoria, ARIN Fellow. This question is actually for Brent. During your presentation, I thought you mentioned something about deployment with EVPN with v6. Wanted to understand, was that for multitenancy or you needed some additional segmentation, what that looked like.

Brent Mc Intosh: You got it. Again, they’re independent protocols. For resilience and high availability and statefulness for the BNGs that are actually doing the DHCPv6 prefix allocation, we decided using BGP and VPN would have made sense.

And as a matter of fact that works like a charm. Once BNG goes down, your IPv6 session with your PPPO is still up and running. And that made because things are sort of replicated between sites. Right? They’re not in the same physical building.

One BNG might be in one location and one might be at another location. We found what was the best way to do it. As a matter of fact, even the vendors were pretty much impressed with the way it was done.

So it’s something to think about. That’s why I say this deployment may be cookie cutter for the other Caribbean ISPs as well.

John Curran: Down to our final question. Final. Go ahead.

Dustin Moses: Hi, Dustin Moses from Intermax, ARIN 51 Fellow. There seems to be kind of a mixed opinion as kind of the future of what a v6 network looks or an IPv6 network looks like. There was the question of we should only be v6 and then there’s the reality of dual stack, right?

And so I guess my question is, when it comes to the rest of the world and all the applications and services that still aren’t v4 and the complications of CGNAT and NAT 6to4, what does the end goal look like with v6? Is v4 always going to continue to be the legacy product that we just slowly ebb off of, or are we really building dual stack networks for the majority of purposes? Or are we trying to strive for a v6-only network and let v4 systems or who doesn’t get on the v4 train basically say sorry, your services are deprecated and you need to figure out something?

John Curran: I don’t know whether to thank you or swear at you, but you literally just stole my closing question. What are we building towards? Are you building towards dual stack forever or are you building towards eventually a v6-only or v6 with gateway to dead v4 applications? Who wants to go first?

Ben Bittfield: I’ll take a stab at it.

John Curran: Okay, go ahead Ben.

Ben Bittfield: Maybe this is my rose-colored glasses from being in the mobile space whereI’m limited in applications in house. But I love the idea of a v6-only environment with limited NAT64 at the edge.

Like obviously there’s going to have to be some level of v4 support to reach v4 things on the Internet. But predominantly, previously when I was at a telco 70 percent plus of the traffic was v6 end-to-end. So why do I need a build dual stack network to support the growing minority of traffic? If I can make everything v6 only great but there will be some limited NAT64 for the foreseeable future.

John Curran: I saw Brian. You were chomping at the bit to answer. Go for it. Are you building for a long-term v6-only or is it dual stack forever?

Brian Jones: We’d like to see a v6 only. That’s what we’re building toward. When we started, we wanted to build parity into the network. So dual stack made sense. We wanted to have v6 everywhere we had v4. That made sense when we started.

Now, we’re looking at pockets of v6 only. We’re trying to do that with our wireless deployments. We’re trying to do that where we can and that’s what we’re building toward in the future. We feel like that’s the way to go.

However, we realize there’s going to be a very long tail on the end of v4.

John Curran: Matthew, what do you think?

Matthew Wilder: I’m also chomping at the bit here. I would say the biggest observation I would make is the Internet protocol happens to not only work for making the Internet work, but happens to be deployed by enterprises for their use cases.

And that’s a key point because in the enterprise space you can actually have a use case where you define that all the endpoints are in your control. The whole issue of that long tail v4 we talked about, that is a general any-to-any case that the Internet has, and you have to have that v4 component for those.

But within enterprise, I think there is some hope you could control more and you can have sort of opportunistic IPv6 only. So that’s probably the golden chalice there.

John Curran: Brent, what do you think? What are you building towards?

Brent Mc Intosh: Very similar to my friend at TELUS. I remember, when we started at this, the service provider world, dual stack was going to be the way to go because at some point we’re hoping that v4 will fall off and it will be a v6-only world, because that’s what we want to do.

But now when I got on the enterprise space, clients are now we see them offloading their hardware and they’re moving into the cloud. Now at AWS and Azure, and they all support v6 -only VPCs, the idea now we’re going to have a secure, seamless connection to these cloud services. And we can do them v6-only.

So while we’re looking for dual stack to be the path to v6 only, we can start doing pockets of v6 to improve the way our customers connect. It’s getting better.

John Curran: Does that help? Does that answer your question?

Dustin Moses: Yes, thank you.

John Curran: I did close the queue, but, Scott, I see you snuck in, and I’ve got three minutes. Come on up. You’re the last, last question. Go for it.

Scott Johnson: Thank you, Mr. Curran. Scott Johnson, SolarNetOne.

Persuant to Mr. Moses’ question, the majority of our mobile devices are now numbered with v6 only and we use a mechanism called DSLite to provide v4 services to that.

In that light, there are large numbers of deployed networks that are v6-only. And given the opinions of the panel here, is it wise to start to consider a sunset date for IPv4? Thank you.

John Curran: What do you think? Do we need a sunset day for v4? That’s even a better end question. Who wants to go first on that one?

Matthew Wilder: Maybe time for a “World No IPv4 Day” or something.

John Curran: A “World IPv6 Only” or a “World No IPv4 Day.” Yes. Other comments from the panelists? Brian, good idea? Sunset day for IPv4?

Brian Jones: I don’t want to be the one to set that. But it sounds like a good idea.

John Curran: Got it.

Brian Jones: It would give people incentive, but I’m just not sure I don’t know if we can do that.

John Curran: Ben, you ready to set a No IPv4 date out there?

Ben Bittfield: Definitely not. I think it’s a great idea, but I don’t want to be the one you’re not going to get anybody that are making money on it.

Matthew Wilder: It’s okay, I said it already.

John Curran: Got it. Brent, what’s your thoughts?

Brent Mc Intosh: I think we should set some timeline to sort of get there. But as Ben said, I can’t make that comment. We should have something saying we should aim to get to that date because there’s tons of customers that’s going to be coming up and don’t have any v4 to get, they’re going to be v6 only anyway, so we probably should set some date.

John Curran: Put some date out there. Good to know. I’d like to give a rousing round of applause for our panelists, both online and present.

(Applause.)

That concludes our session. Thank you, folks.

We’re now going to transition to break. I’m going to bring it back to Hollis. Go ahead. Thank you, guys. Here’s my virtual handshake for you guys. Thank you.

Hollis Kara: Thank you, John, and thank you panelists. That was great. All right. Whoops, getting ahead.

Yes, we’re going to take a break now. We will come back at 3:15. So it’s going to be a 30-minute break. And if you look at your agenda, we are slated to start back with the RIR reports. We’re actually going to slide in one presentation that got bumped this morning before we do those. But we’ll still go through the rest of the items on the agenda before we close today, just so you know.

When we come back, we’ll have Leslie Nobile up on stage to give an update on her work. And I look forward to seeing everyone at 3:15.

Trust and Public Safety

Hollis Kara: It’s 3:16. We’re going to start. Welcome back. We are in the homestretch. Folks will be scurrying in, I think, over the next few minutes and pretending that they were here already.

We’re going to kick off this afternoon’s session with, the last session of the day, with Leslie Nobile, our senior director of trust and public safety, to give a trust and public safety activities update. So Leslie, if you’d like to come on up. Getting a sip of water. They’re rolling in.

Leslie Nobile: Good morning. Sorry, I was supposed to be speaking this morning. Good afternoon.

My name is Leslie Nobile. I’m going to give you a quick update on some of the work I’m doing in the trust and public safety area.

So okay. So I get this question often, and I thought I’d throw it out there because some of you might wonder why does ARIN support law enforcement and public safety; why are you even working with them? They’re not part of the industry.

But there’s a myriad of reasons. Collaboration and information sharing between ARIN, law enforcement and other trust communities can do a number of things. First, it does support ARIN’s mission of helping the Internet function in an open, stable and secure manner. Obviously, we’re working with law enforcement. That’s what they do too. So this is very important stuff.

Our work together provides them with relevant information and tools that can help them in their investigations. And the tool that comes to mind first is Whois. Law enforcement often goes to Whois at the very beginning of an investigation to look for who might be using an IP address. They’re also looking for victim information sometimes.

And one of the main things they’re looking for is jurisdiction because they operate within boundaries and borders. So they’re looking to see where the jurisdiction is.

So another thing that we do is, with our work, is we’re contributing to resolving fraud, abuse and other cybersecurity-related incidents. And there’s many instances of law enforcement coming to ARIN and making requests with court orders, subpoenas, even just questions, just phone calls to get information about who might be using IP addresses and how to interpret things and all kinds of information that they need to solve cases.

There’s also instances of ARIN going to law enforcement and bringing fraud cases to law enforcement. And one that comes to mind is Micfo, which was successfully prosecuted. And that was done by ARIN when we discovered fraud. So there’s a lot of back-and-forth collaboration.

And finally, our work together promotes the multistakeholder approach to Internet governance. And you’ve heard Einar and you’ve heard Bevil. What we hear everywhere is multistakeholder collaboration is imperative to the success and future and growth of the Internet. So this is part of ARIN’s mission as well. So we are players in there.

So how do we engage? What does it actually mean? We collaborate. We work together with our industry and government partners.

We participate. We participate in all the relevant venues and forums.

We do outreach and education. That’s a big part of my job, in fact.

We facilitate. We facilitate conversations. We introduce people. We do a lot of information sharing.

And finally, we cooperate. We cooperate very closely with law enforcement. We respond very quickly to subpoenas and court orders. And Michael and Steve will even work with law enforcement on some of those so that we’re getting the right information and they’re getting the right information in turn.

These are examples of stakeholder communities that I engage with pretty heavily. There’s sort of four areas. Much of this is active participation with these stakeholders. And some of it is active monitoring.

So law enforcement and public safety is all active participation. These are examples of some of the groups and organizations that I engage with pretty frequently, FBI being at the top of the list, quite honestly. Homeland Security — actually I’m putting together a training course for them within the next couple of months.

Our Canadian partners, the RCMP. Interpol and Europol, quite a bit of engagement with both of them. And then we’ve got our civil law enforcement folks, the FTC, the USFTC and the Canadian CRTC, doing a lot of collaboration with both of them.

We’ve got our intergovernmental organizations that we engage with, and the UN being sort of at the top. And that’s active monitoring. Not a lot of participation there.

The Internet Governance Forum, we’re very active there. We run a booth for the NRO, the five RIRs together. The Caribbean Telecommunications Union, you’ve heard about already. We do support them.

Our industry partners, a lot of collaboration with them. RIRs and my counterparts at the other RIRs, we form a group called te Public Safety Coordination Group. And we do a lot of collaboration and information sharing together.

I do a lot of work with ICANN and I do a lot of work with my tongue twister here, the Messaging Malware and Mobile Anti-Abuse Working Group. It’s known as M3AAWG for short. I’ll talk about them in the next slide.

NANOG and the DNS Research Federation, you heard from them this morning. They have an ARIN grant, and I’ve been working with them for over a year before they officially launched in January.

And then finally, there’s a lot of different cybersecurity groups that I engage with, the GFCE, Global Forum on Cyber Expertise. We had the president, Chris Painter, as our keynote speaker at an ARIN meeting maybe a year and a half ago. Some of you may have attended.

The NCFTA, that second one, that is an alliance between the FBI and industry partners. They share the same offices and exchange information. They work together on cases. I do a lot of training with them, in fact, and a lot of collaboration. A lot of phone calls, a lot of information sharing.

And the Canadian Cybersecurity Center, I was up there training, I think pre-COVID when all the good stuff was happening pre-COVID. COVID put an end to a lot of it. But we’re starting to see a bit of an uptick in this, but I’m hoping to get back up there at some point.

So I wanted to mention a couple of groups that I particularly focus on. Law enforcement, all the stuff you heard me say. Outreach, education, capacity building, information sharing referrals. I get a lot of phone calls from them saying who can I speak to at another RIR? Who can I speak to in industry? I have a question about this. I Act as a facilitator.

Policy Proposal guidance, we have actually had cases of law enforcement and public safety organizations submitting policies at ARIN.

We had the DEA, the Drug Enforcement Administration, submit a successful policy. We’ve also had the FBI submit a policy that was passed by the community.

And I believe there may be more policies coming from their direction at some point in the near future.

And then Whois, of course. There’s lots of questions about Whois, how to use it, bulk Whois, historical Whois. So lots of work in that area.

M3AAWG. It’s a global anti-abuse industry association. And their main focus is industry collaboration in the anti -abuse fields, obviously, spam, et cetera, and producing BCPs.

They really are heavy duty spam fighters, serious security geeks. But they had a committee called the DNS Working Group. And they decided to integrate IP addresses because Internet identifiers are what make the Internet work.

So they upgraded the working group to a committee. They added numbers and they asked me to be the co-chair. I’m doing quite a bit of work in the M3AAWG space, bringing information about the RIRs to M3AAWG because there’s been quite a bit of misinformation and misperception from the M3AAWG community.

They really don’t know much about us. So part of my role is bringing that information to them.

They’re very interested in partnering with the RIRs. In fact, they consider us one of their partners.

They run a public policy committee that is awesome. They focus in three areas. They have Europe. They have Canada and they have the U.S. And so we get updates biweekly on what legislative activity’s going on and who is doing what.

Then I can bring some of that information back to my team, the Government Affairs Department. So it’s been really good participating there.

Some of the topics that I brought into M3AAWG from presentations at their meetings — they have three meetings per year — we did, about the NRO to tell them who we are and what we do. We did one on RIPE labs and RIPEstat, really good information for the researchers in the anti -abuse community.

We had Brad Gorman do an RPKI fundamentals talk. We had Lee Howard do a talk on IPv6. And it turned out, a lot of the discussion went toward IPv4 depletion, the IPv4 market, fraud in the IPv4 market, leasing in the IPv4 area and the lack of attribution in Whois because people are losing track.

Sometimes when space is leased, it doesn’t get recorded in any of the registries. So that is a is future talk. We’re proposing that a panel at a M3AAWG either this fall or next February, I’m not sure when that will happen.

And finally, ICANN engagement. Sorry, I’m dry. I do a lot of work with ICANN. My partner, my colleague, is Carlos Alvarez at ICANN. And he is their global law enforcement liaison. He does a lot of work in the DNS area, DNS abuse. He supports law enforcement all over.

And we were both doing separate presentations for law enforcement and realized that there was a lot of, sort of, misinformation. They didn’t really understand the difference between domain names and IP addresses and where the distinction was. And so, we decided to join forces and do joint presentations.

So we started doing a lot of the FBI field offices, Interpol, Europol. Just a lot of things. So we’re doing this ongoing training outreach capacity building together and that’s been great. In fact, he’s going to do the Homeland Security training with me this summer.

ICANN runs law enforcement training workshops, two-day workshops at every meeting. And they rely on the regional law enforcement agency to sort of coordinate with, and they have asked the RIRs to participate in that for probably the past 10 years I’ve spoken at those.

We just go in for a couple of hours and do a session. We don’t attend all day. That will be happening in Hamburg, I believe, in the fall.

And ICANN also does How it Iorks training sessions. And typically they’re about domain name abuse and about DNSSEC or about some ICANN tools. But in Montreal, I think, just before COVID — everything was pre -COVID — they asked the numbers community to come in do an update and training.

I put together a presentation and did that with Paul Wilson from APNIC. And I’m told that they are going to reinvigorate that program this fall, and they’d like the numbers community, they would like us as the the RIRs to come in and participate. So those are some of my ongoing things.

So you heard Einar mention we do some monitoring. We look at various things for potential impact to our ARIN community. So in the geopolitical, legislative, and regulatory activity, I’ve been looking at a couple of things.

Einar told you a lot about President Biden’s US national cybersecurity strategy. It’s actually putting more of a burden and more responsibility on vendors and on network operators and you guys, really, to make sure that your infrastructure is secure.

And so there’s no legislation right now, but there could be. So we’re watching to see if anything comes out of that, anything that could really impact our community.

The second thing I’m watching, and to me, it’s actually even more interesting because there’s an existing law, legislation called the Computer Misuse Act of 1990 in the UK. The UK decided that they needed to empower law enforcement and give them more teeth to fight cybercrime.

And so they put out a consultation and they want input from industry. And there were two questions. And they were directly related to the seizure, allowing law enforcement to seize domain names and IP addresses.

And it was very broad questions and a little bit scary because I don’t think they really understood the implications. So I’ve spoken with my RIPE NCC colleagues about it. And M3AAWG ended up providing comments, very good comments on the entire consultation.

But regarding the seizure of IP addresses, they basically said this is overreaching. Well, they said it in a nicer way than that. This is very broad. This will have unintended consequences, and you really need to consult with the five RIRs before you go any further. So kind of following that to see what comes out of it.

Speaking of RIPE NCC, they put together a database task force to look at what information is required in Whois. And their community basically decided that there’s certain things that are not really required. They may not need to be in there any longer, so or they may become optional.

It’s information that network operators rely and law enforcement relies on. I’ve been really watching this. The recommendations have gone to their working groups and there may be policy proposals coming out of that. I am focus edon that. I’ve informed my law enforcement colleagues. And we’ll see what happens.

Then finally, I mentioned the UN. This is strictly monitoring. There’s two groups dedicated to cybersecurity. There’s The OpenEnded Working Group and there’s the Ad Hoc Committee on Cybercrime Convention.

And basically it’s global cooperation amongst member states to establish cyberspace rules. They’re working on resolutions, they’re coming to common ground. Basically it’s going to allow law enforcement to do, to fight cybercrime across borders. So we’re following that to see what comes out of that.

There’s a hyperlink on all of those, if anyone wants further information on any of that. You can get it right off of the slides.

My final slide. I’m closing with this, because this particular quote really resonates with me. I was reading Tech Republic. And it was about the new national cybersecurity strategy. And an industry expert said, “Intensive and open collaboration of independent experts coming from industry, academia and specialized organizations would help by producing balanced regulations amenable to both industry and government.”

And this, I feel like this is part of our mission at ARIN, particularly in the Government Affairs Department. So this I wanted to close with.

And that’s all I have. Are there any questions?

Hollis Kara: If you have questions for Leslie, please feel free to approach the microphones or start typing. Give folks just a moment. We like to give them a second to type.

Leslie Nobile: I want to run.

Hollis Kara: Leslie is ready to run. Type fast.

Edward McNair: Edward McNair, NANOG. Just a quick, simple question.

I know there’s a lot of ARIN information that’s public. And then there’s stuff that’s not public. When do you guys make that mandate? When law enforcement comes, do you quickly give them the public information? What’s that line drawn where you give them the private information?

Leslie Nobile: You answered it yourself. Whatever is publicly available on Whois, we talk about that, we tell them how to interpret it. And whatever is not available publicly, we require a subpoena or a court order if they want us to take further action.

Edward McNair: That’s what I thought. Just didn’t want to assume. Thank you.

Leslie Nobile: Anything else? Thank you for your attention.

Hollis Kara: I think we’re done. Thank you, Leslie.

(Applause.)

All right, next up we’d like to feature updates from our four brother, sister, sibling RIRs, if you will. We’re going to start off with an update from James Chirwa from AFRINIC. That is not what that is. Jeez, this is hard guys, I’m sorry. It’s a long week. Ok, there we go. All right, roll tape.

Regional Internet Registry Updates

AFRINIC

via video

James Chirwa: Good day to you all. I believe you are all having a wonderful event so far. I will be giving you an update on AFRINIC operations, and it’s always a pleasure to join the ARIN community.

Unfortunately, today, I could not make it for an onsite presentation.

My name is James Chirwa, Head of Member Services at AFRINIC. On today’s agenda, I will bring you the AFRINIC strategic pillars for the current strategic plan. I will also provide you updates on our registry services, policy development, technical projects, capacity building and other outreach engagements.

We are currently in the final year of our three-year strategic plan. The plan has four key pillars driven by 15 focus areas.

The first key pillar is engagement, focusing on enhancing engagement with all relevant ecosystem stakeholders.

The second pillar is service delivery. That strives to ensure good customer experience through innovation and research in building our new products.

The third pillar is operational excellence for customer business model optimization, agility for efficient registry function and technological enhancement.

The last pillar but not the least is organizational performance through improved organizational culture and structure that ensures a more people and talent centric organization. Further down the line we also focus on improving the financial position of the organization.

These four pillars are condensed in the following core values and the vision and mission statements.

Now I’ll move on and speak about the registry services. Currently, AFRINIC is serving over 2,150 resource members across 55 economies. This represents roughly 6 percent growth compared to the previous year at the same time right now.

To this and Number Resource members, this year alone we have allocated at least 56,000 IPv4 addresses, bringing the total of our dedicated IPv4 resources to over 115 million IP addresses. This number represents over 95 percent of our entire IPv4 pool.

In IPv6, we have issued 24 prefixes this year, various sizes. Approximately we have issued roughly 10,000 of IPv6 /32s.

And now on Autonomous System Numbers, we have issued 49 new Autonomous System Numbers this year, bringing the total to 2,285 of issued Autonomous System Numbers. This is 7.3 percent rise compared to last year.

Now as we are nearing exhaustion of IPv4 in our region, we have currently have about 1.3 million IPv4 addresses available in our pool. And the maximum delegation at every instance is capped at a /22, based on the provisions of the Soft Landing Policy phase 2.

Even though there is a limitation of /22 at the time, the member can come back for more upon reaching the required threshold of 90 percent and they have justifiable needs.

Now I will move on and provide an update on the Policy Development Process. We have a number of policy proposals at different stages. The first three are awaiting Board ratification. They have already gone past consensus.

The first one is the Number Resource Transfer Policy Inter-RIR. The policy allows transfer of IPv4 to and from the region. However, the resources moving out of the region will only be legacy or resources received through another transfer prior, another Inter-RIR transfer before.

The second one is the Abuse Contact Policy Update. This one seeks to introduce a mandatory abuse contact in the AFRINIC Whois and further provides a framework to ensure currency and reliability of the abuse email address provided so that the network providers are able to delete that in a timely manner in case of any abuse.

The third policy is the Policy Compliance Dashboard. And this one seeks to improve the compliance of users but also make our policies more effective.

Through our technical — now I’ll talk about technical projects. T hrough the technical projects, we have launched a new datacenter in Mombasa, indicating a different geographical location compared to our current, to where we hosted most of our services This has improved our level of redundancy in our service offering.

We have also undertaken other changes like cloud migration on our collaboration to Whois, moving from on premise. Currently, we are also working on improving the user experience through implementation of the resource management portal. We are also improving the graphical user interfaces and functionalities for services like RPKI and IRR.

Now I’ll move on and talk about capacity building. In the previous year, we made some strides in capacity building through technical webinar series where we undertook two webinars focused on RPKI and IRR with at least 96 participants. We further also did a lot of work on deploy-a-thons where we enabled at least 55 organizations, attained 95 deployment milestones towards adoption of IPv6, RPKI and IRR.

Through our e-learning academy, that has on-demand training modules for Whois, IPv6 implementation, IRR, RPKI, which are offered in three different languages, we have seen over 3,000 enrollments over the time. Furthermore, we’ve also been conducting IPv6 deployment help desk, where we are able to handle at least 93 support calls from 59 organizations.

In this year, we are still conducting capacity building through trainings, trainings online, face to face, and also through direct partnerships with universities across the region. We are also still focusing on deployment support through the deploy-a-thons, dedicated help desk for IPv6 deployment, as well as webinars.

In addition to that, we are also offering IPv6 certification exams for our gold and silver levels. Further, beyond the capacity building, we’ve also engaged with our various stakeholders in different activities.

Last year, we undertook engagement activities with various stakeholders through our annual African Internet Summit event that took place from the 30th of May to the 3rd of June with over 500 individual participants attended.

We also held a special members-only engagement event in November, with over 200 members who participated to connect and share different technical knowledge and experiences.

This year, we also expect to hold the annual African Internet Summit and General Members Meeting as soon as all enabling provisions are back in place. In our further extended engagement with other stakeholders beyond the resource members, we also gave presentations at other events like the Internet Governance Forum and technical events.

We also had collaboration with government agencies and other partners on strategizing towards IPv6 deployment as well as projects on enhancing Internet critical infrastructure like IXP and DNS and other projects for DNS resilience. We also provided program support that targeted over 2,500 beneficiaries.

This year, we continue to undertake in some of these engagements, though at the moment mostly remotely. We expect to undertake some at full scale as soon as possible.

That’s all I have for you today. And thank you all for your attention.

(Applause.)

Hollis Kara: Awesome. Next up, Karla Skarda, Services Director from APNIC, with an APNIC update. Hi, Karla.

APNIC

Karla Skarda: Thank you. I brought my timer with me to make sure I stay on track. But while the slides are coming up, I just wanted to give a shout out to James and all our friends at AFRINIC.

I think it’s an incredible job that they’re doing, that they’re still managing to keep going at the pace that they have. So congratulations to them.

(Applause.)

I’ve just realized, I’ve got control. Look at that. So I am just going to run you through a few things here. So the first thing is to just let you know that I’m from APNIC. My name is Karla Skarda. I’m the Services Director there. This is the region that we look after. As you can see, it’s rather massive.

And we were talking about how some of the different economies that we look after, there are 56 in the region that we look after that are very diverse. So we’ve got a really wonderful, diverse team that’s based in Brisbane as well. And I think we have about 36 of those 56 economies actually represented in our company.

So most of you already know this, I’ll zoom through this here, we have what’s called the National Internet Registries that represent APNIC in some of the region that we look after because it is so large and so diverse in terms of politics and also language and culture.

So that’s something that’s fairly unique to us and also our brothers at LACNIC.

So just moving on to some of the stats. We currently have a membership of just over 9,400 direct members and just over 20,000 if you include all the members that are part of the NIRs.

That’s the two charts there. The bar chart is showing you the various different NIRs that are represented there in terms of membership numbers. And then the little pie chart shows you the different economies that we look at. And these are available online. So I’m going to scoot through these for you to just get a sense of the regions that we look after.

IP address delegations, over the last 10 years, as you can see, I’ve got IPv4 and IPv6 there. It’s kind of been a little bit, probably quite stable over the last two years. We found that there has been some challenges around COVID that we’ve continued to have some issues with.

But also IPv4, I think we’ve probably been [unintelligible] ourselves and AFRINIC are the last two of the RIRs that still have IPv4 address space that we can delegate. So we’re very fortunate to be in that position.

Resource delegation by region over the last 10 years. That is, you can probably see that it’s gone, if you look at East Asia it’s gone down dramatically over that period of time. And if you look at South Asia, that’s the region that seems to be growing a lot at the moment.

IPv4 transfers, everybody’s very interested in this. I’ve got the three different slides up here. So the first one is the resources that we transfer out of the Asia Pacific region. It’s showing you where it’s going to internationally.

Then the second one there is the transfer within the Asia Pacific region, inter-region. And then the last one is the one of IPv4 transfers from different regions and these numbers are in /24s.

So I’m going to zoom in a little bit on to the ones that I’m sure you’re more interested in. As you can see, by far the largest region where we interact with is actually with ARIN. So I’m just trying to squint at these numbers here, but I think it’s about 71,000 addresses, address blocks in the last 10 years that have been transferred from ARIN into our region — or the other way around. Sorry.

Moving on to a topic that seems to be quite topical today, which is the ROA or RPKI deployment in the various different regions. So this gives you a little bit of a heat map. And, again, we have a nod to RIPE and RIPE Labs for providing us with this information.

And as you can see, there’s a huge variety of the deployment status of ROAs in the different regions. So we had a chat — we have the opportunity of interacting with our members when we do the annual survey or, sorry, it’s conducted every two years, not annually. I wish it was. And we get some feedback from them as to what some of the challenges are.

So of those 3,000 odd people that we actually had interviews with, 60 percent of them in the APNIC survey indicated that the organizations had not deployed RPKI yet.

So that gives you kind of an overall perception of what’s actually happening in the region.

38 percent said they didn’t have the knowledge or expertise to deploy it. So that means we’ve got a lot of work to do as an information center to continue to work with the training team to get the knowledge out there.

And then the third group, the third point there that was really interesting for us was that the registry requires continued investment to meet the architecture, availability and robust requirements of RPKI.

So that was really an invitation to us to continue to do what we’re doing in those areas of development. So that was a really good green light for us to go, move further into providing the infrastructure that we need to support these important projects.

So having a little drill down into IPv6 uptake in the Asia Pacific region. So the world average is 38 percent, apparently. And so of the economies that we look after, eight of those economies are just above the average. And nine of those are significantly below the average.

So, as you can see, it’s a broad range of different challenges that we need to deal with.

But I’m very proud of those countries that have actually made it above the line. Australia, which is where our offices are based, are not quite there yet. But we are moving forward with that.

Moving on to the policies. So we just had our conference in the Philippines in February — February and March. And at that meeting there were three policies that were proposed and reached consensus.

The first one is around the Historic Resource Management Project that we started in 2022, which is very close to my heart because that ended up being a project that I had to really look after that was in my stable.

So one of the challenges we had was that once we had recovered those addresses we didn’t have a mandate from the community as to what to do with those and how long we needed to keep them in quarantine, so to speak, before they would be available.

So that policy reached consensus and the policy has been said that those need to be placed into reserve for a 12-month period before they can be made available. So that’s in line with our current policies, which has been really helpful for us to get that direction.

The next one was about ROA objects within private, reserved and unallocated space. Just basically advised that it’s not necessary to create those ROAs in that particular space. There was some guidelines that provided for this before the policy was actually brought into effect. So this is really just to cement those guidelines in place. Ooh hello.

Hollis Kara: That was me, sorry

Karla Skarda: Just thought the aliens might have come down.

Then Prop 151 was the restriction of the nonhierarchical assets, which is also going to be quite helpful in making sure that that recommendation is followed.

The one policy that we seem to kind of never actually get over the line that is proposed at pretty much every meeting is about changing the size of the blocks that we delegate. So that one did not reach consensus. There was actually a technical issue with how that was presented.

But it is something that we are continually on the edge of our seats waiting for that to come through. There’s more information in the link I’ve provided there below to give you more details on those policies.

Now moving on to the activity plan that we have at APNIC. So we do follow an activity plan. We are reaching the end of a four-year strategic cycle, but all of our activities are in these pillars. Those first two pillars, the membership and registry pillars, are mostly my responsibility, together with a wonderful development team who are just really very supportive.

And obviously we’ve got, because we are an information center as well, we’ve got a strong focus on development information which includes training as well. And also capacity, which James spoke about as well quite eloquently.

Some of the highlights of some of the things that we’ve been able to achieve. I’ve put in 2022/23 because we kind of are covering a little bit of end of last year as well since the last time we actually were able to present here at ARIN. So the APNIC survey, as I said, that’s a very, very important piece of a collection of information that we get from our participants.

So, again, there’s a link there if you want to go and see the actual results of the survey. Some very interesting stuff that came out of that.

The Historical Resource Management Project, I’ve got a slide specifically on that. I’ll take you through a few details because again, I know that’s something that a lot of people have asked me about here since I’ve been here at this conference.

There is a new MyAPNIC dashboard, which has — it’s all got a big overhaul and that’s ahead of a much bigger project that we’re starting this year as well.

Route management prevalidation, which is quite important as well, particularly when it comes to transfers. We’ve also launched a new community platform where we are trying to move away from Mailing Lists and starting to invite a more dynamic conversation online. So please feel free to check out Orbit and contribute if you feel like you’d like to get involved.

There was also over 30 tech community events that we have already attended this year. It’s gotten off to a really strong start. And 22 NOGs where we’ve either provided sponsorship, speakers, training and technical support. And that’s been a pretty heady start to this year already.

So this is, as I said, I’m just going to brag about this a little bit because it is a project that has been sucking up a lot of our time and resources. So we started off with just over three and a half thousand cases, which was really just these were the objects that we had that we kind of knew that there was somebody that had that space.

That’s where we started. There’s been a lot of consolidations and things we’ve discovered along the way. 7.3 million IP addresses that are affected by this project.

So at the start of this project, we had slightly later start than what we hoped. We started the project in March of last year where we had, of those — you can see the different, sorry, there’s a little bit of a blur in the middle of the page there. Sorry, bad alignment.

But we started off with those three and a half thousand members yet to contact. And you can see the progress over the period of time. So we actually managed to contact what all those address holders at least once by the end of last year.

So we sent out emails. If we didn’t have an email address, we had people frantically searching everywhere they could find to try to find them. So I’m really proud of the team that they were able to achieve that by the end of the year.

We are now sitting at a point where we’ve got about 500 that are in work. So 500 resource holders have indicated that they’d like to claim those resources. So we’re busy working with those.

But we also have still about 800 blocks that we have not been able to contact anyone. They’re all unrouted. So that’s something that we will be churning through the rest of this year to try and see if we can bring that to some sort of resolution.

But there is, again, a link in there if you want to know more about how that project has actually been going to date.

And that brings me to my last slide, which is just talking a little bit about the roadmap as we are moving forward into Q2 and Q3 this year. I would say the MyAPNIC dashboard was part of the beginning of a total website refresh that we’re doing.

We’ve had our UX designers clawing their way through and pawing little fingerprints through the systems to see how the systems are being used and where the blocks are. So we are in the process of that’s going to be properly taking us well into 2024.

And then we’ve also got the alternate Whois authorization. Up until now it’s only been a corporate contact. So we’re changing how we’re actually managing our contact database completely, so it’s a total revamp.

And together with that we’re also going to be introducing two-factor authentication mandatory similar to what ARIN’s done recently. So that’s another hopefully that’s going to be completed by June 2023.

So there is also something I’m very excited about, which is the registry API that I’m hoping is going to give us the opportunity of increasing the accuracy from all the various different ISPs.

What else have we got? Changes to account contacts, I’ve already mentioned that. And the implementation of a new payment gateway. We have a lot of challenges with different economies, some of them, because of the exchange rates, have a lot of challenges actually paying in Australian dollars and have to chunk down those payments into smaller pieces. So this is going to make their lives a lot easier.

I don’t know why I have this again. I went backwards. That’s why. I was just showing off. And that’s me. Thank you very much.

(Applause.)

Hollis Kara: Thank you, Karla. Did anyone have any questions for Karla before we trade out? No.

Thank you, Karla, and, again, my apologies for the disruption. It wouldn’t be a meeting if I didn’t try to create an international incident.

Next up, I’ll invite Alfredo Verderosa from LACNIC to share what’s happening down there.

LACNIC

Alfredo Verderosa: I’ll share a few slides which focus on what has happened with Internet number resources in our region after exhaustion a couple of years ago.

And I’m going to talk about requests, waiting list, resource transfer, some relevant policies and a little other information.

About IPv4 exhaustion, it was on August 2020, you can see that in that quarter we had twice the number of requests we usually have. It was about a thousand requests.

And after that, the interesting thing is that we were expecting to return to levels below exhaustion. But we remained at the same levels, actually a little above those dates — that field, sorry.

Our membership base continues to grow at a smaller pace but it’s still growing. Last year it was plus 5 percent. We reached 12,500 members. And right now we have about 1200 IPv6-only members.

So what happened with members that request IPv4? They enter a waiting list we created in September 2020. Since 2021, I believe it was in April, all the members, if they want to, sorry, in order to enter the waiting list you must be a member. So you need to sign a contract, request IPv6 and then you enter the IPv4 waiting list.

This waiting list has been increasing a lot. Now we have more than 1,000 organizations on it. This is because we received from 40 to 60 requests every month. And we can only fulfill maybe four or five. We’ll recover only a few IPv4 every month.

So the last on the list, by now we’re estimating will get IPv4 by 2030, 2030. It’s not a mistake. That’s the date.

And that’s a lot of time even compared with the 800 days that the organization that they’re receiving right now IPv4 helping waiting in the list.

Now, talking about IPv4 transfers. We have a low number compared to others, but they’ve still increased the last year. Last year was 145 transactions, with maybe 25 of them in the RIR. Most of the other RIRs, interRIR transfers are with RIPE. Sorry, I’m sorry, the slides are in Spanish.

Hollis Kara: I didn’t even notice that.

Alfredo Verderosa: Well

Hollis Kara: I didn’t do that.

Alfredo Verderosa: The net result of the IP movement per country, the biggest gainer was United Emirates and France. And the biggest not gainer, I would say losers but the countries offering more IPs were Brazil and Chilé.

It is important to mention that there were some big intercompany transfers between those countries. And then I’m going to talk briefly about two policies. One of them is the Point of Contact Validation Policy that was — it is a policy we implemented two years ago. But now we are finishing the first cycle. We have already validated 99.9 of the Point of Contacts. We have started with more than 3,000 two years ago. The first step was following a campaign. After that, we started with automated emails, sending twice a month emails to the Point of Contacts after they validate.

By November 2021, with those that didn’t validate, we blocked MyLACNIC access. It was very effective because in only two more months we were able to validate 700 organizations. And in early 2022, we initiated an additional effort with the 300 remaining that was phone calls and looking for additional contacts on websites.

And February this year, the 20 remaining organizations started the revocation process, and I believe in May we are going to revoke maybe five of them. The others have already updated.

Okay. Another policy I would like to mention is one that we are implementing right now. It is in order to authorize recipients of delegated blocks IPv4 to IPv6 to create RPKI ROAs in hosted mode without the need to ask the resource holder. That is going to be very soon.

Finally, I want to comment that our last year we have our Customer Satisfaction Survey. We achieved 96 percent of members in the top [unintelligible] answering four out of five. We’re very happy with that because it was a challenging period. We were initiating, it was from 2020 to 2022 that we have IPv4 exhaustion. Also, the COVID in the mix. So we’re happy with the results.

And I’d invite you all to our meeting. It’s going to be held from 8 to 12 of May in Merida, Mexico. It’s hybrid. So you can attend in person or online. Thank you.

(Applause.)

Hollis Kara: Thank you so much. Walk here, walk there. You’d be amazed at how many steps I’m getting in.

Marco, you’re already up here. Come on. Update from RIPE.

RIPE NCC

Marco Schmidt: Good afternoon, everybody. I’m Marco Schmidt, Manager of Registration Services of RIPE NCC. Glad to be here and glad to give you an update from our service region. I’m aware about the time. We’re a little bit behind, and I think I’m one of the people that stand between you and a sunny afternoon. So I’ll try to keep it short.

First off, a bit looking forward. We have actually published plans for this year in December. I will not bore you with the numbers. Whoever is interested, there are links included in the slide deck. You can look it up. But basically our costs went up by about 10 percent, mostly related to inflation, and so did also our service fees.

And our activity plan is in line with our five-year strategy that we announced December 2021.

In 2023, this year, we want to focus on a couple of topics. One, the most important one is actually that we want to remain resilient and also customer focused in a time of quite some political and legislative challenges and changes. Just to name one thing, the war in Ukraine, which is in our service region, is putting quite some challenges to us.

We plan to develop RPKI further. There’s an extra slide later. We want to remain a center of excellent data and measurement and tools for everybody who is interested in the development of the Internet.

We need to maintain and to also extend our level of due diligence, our security and compliance, while still trying to provide great service to our customers.

And looking inside our company, happy employees produce happy customers. So we focus on maintaining a healthy organization culture where all employees are aligned and motivated with the organizational values.

What we have been working on so far, a couple of topics. The first one is the registry. So my area. We currently have, like LACNIC, a waiting list. And right now we have around 1,100 LIRs on that waiting list.

You have to be a new member or don’t receive any address space before from us to be eligible on that waiting list.

Currently, if you would join the waiting list now, you can expect you have to wait for around two years. So not as bad as LACNIC, but around two years before you can get a /24 allocation.

We have strengthened our efforts in regards of due diligence and compliance. We have now automated sanction checks because we have to make sure, as we’re an organization under Dutch law, that we apply with EU sanctions. And as we have quite some countries in our region where organizations are sanctioned, I’m just naming, Iran, Syria, Russia, we must make sure that we don’t violate those rules.

And also we enhanced our due diligence. And if we require, for example, recently more often, notarization.

And all this, we still want to put extra attention, extra support for our members in distressed areas, for example, Ukraine, I mentioned, but also Turkey, there was a terrible earthquake a couple of months ago. And those members, they have other problems other than paying their membership fee. So we tried to make it possible and then accommodate it, so they don’t have to pay now but also don’t face the risk of closure while they’re busy rebuilding their networks.

RPKI, as I mentioned before, is one of our focus points in 2023. We are working to make it much more resilient by better geographic distribution. We are developing a better user interface and also doing an audit, getting an audit done on us to ensure compliance, and also plan to report on that.

And together, with the other RIRs, Karla mentioned it already, we’re actually also working on a more unified approach to ensure a good maintenance of the RPKI data.

We have a very active e-learning, not just only e-learning but our learning and development department is very active and they have quite some e-learning activities and also certification.

One, it is quite new. It’s about BGP security. If you’re interested, you can take this exam. And also online, you can find some new content on our RIPE NCC Academy, which is the platform for all this e-learning material.

You can find something about BGP security, of course, IPv6 fundamentals, Internet governance and also we have some shorter ones, for example, about RPKI and we call those segments, they’re just 20 minutes, microlearning. You can find them on our website.

RIPE Atlas, I think it’s quite known, it’s a worldwide measurement networks with many probes all around the world. We basically are maintaining that network and still expanding it in some areas of the world where we would like to have a bit more coverage.

The United States, we’re very well covered with probes, but, for example, if you happen to have good contact or happen to work in, let’s say, Latin America or Africa, then you might be interested to become a sponsor and help us to distribute probes in those regions.

If you’re interested, there’s a link included in this slide deck. We also published some Internet Country Reports. By the way, I noticed [unintelligible] the whole formatting, it’s a bit off here. I hope you can read it. These are reports done by our research and development team. They look at a certain region and measure all things around the Internet, how is the routing going, how is the DNS, market landscape and so on.

You can find those reports on our RIPE Labs website and not only those reports but also much more information, articles about the latest developments in the Internet and new services by RIPE NCC and so on. And last year we also have launched a podcast service there.

So if you’re interested, labs.RIPE.net, you can find those information.

Another new service is the RIPE NCC Forum. That’s something similar like in APNIC, what Karla mentioned. So that we’re looking for ways for a more interactive engagement with our community and also between community members, not just the mailing list, but to have a place where you can go to our website and you can have some informal discussions about anything around the Internet, our service region, the RIPE NCC services, and it’s open for everyone.

So if you’re interested, again, I invite you to go to our website and you can find this RIPE NCC forum there.

Something else we’re quite proud of is we extend our language support. Here in the ARIN region, you’re quite fortunate that most of your members are from English-speaking countries, the vast majority.

Our region spans from Greenland to Kamchatka and from Norway to the Middle East. So we have 72 countries in our service region and about the same amount of languages. And we did notice that in some areas, in some countries, sometimes people, the level of English there is not as good. And so we try to help them, and we have, with the help of some of our colleagues, created some information on a language wiki page. It’s currently available in Turkey, Farsi, Spanish, Italian, Russian and Arabic, and you can find some information, there are people who speak those languages. How I can become a member? How I can request a transfer? How does the RIPE NCC work? And we also add some Internet country reports there.

Our deck is completely misformatted. But let me just talk about it. We have our regular RIPE meetings. There comes a slide soon. But because of our vast service region, we cannot expect everybody makes it to our RIPE meetings. Like ARIN, you have special outreach activities in the Caribbean. We do this as well, especially in the Middle East. We have a couple of activities there. Like some RIPE NCC Days in Uzbekistan will be coming.

We have a Peering Forum that we did last year in Kazakhstan. And will do this year again. And also some local activities in the Balkan area, for example.

Another interesting information that might be useful for you, ARIN has the ARIN Community Grant. We have the RIPE NCC Community Project Fund. Basically 250,000 euros that we give to projects that are for the good of the Internet.

And I just looked up the deadline to submit your project is in about one month. So if you have something, a project going and you think it would fit those requirements, one of them is that you must have a relation to RIPE NCC region. Then, yeah, I encourage you to submit this application within the next month, basically.

And then I see two slides. Then, of course, I want to invite you in about one month to our RIPE meeting. It will take place in beautiful Rotterdam. By the way, if you plan to attend in person, I recommend you don’t look for flights to the airport of Rotterdam. Rather, fly into Amsterdam and take a fast train there. And, of course, this is the hybrid meeting, so you are very welcome and we’re happy to see you online at our next RIPE meeting.

And that is the end of my presentation. Are there any questions?

Hollis Kara: Any questions for Marco?

Thank you.

(Applause.)

It’s been a minute since we’ve been able to bring back a representative from IANA to give us an update on what’s happening there. I’d like to welcome David Dong to give us the IANA Number Services Update.

IANA Number Services Update

David Dong: I’m David Dong from IANA. Give you a very quick update.

Starting with metrics. Last year, we updated our reporting metrics on our website. We’ve consistently met our Service Level Agreements across all areas in numbers, naming, and protocol parameters. And numbers, in particularly, we’ve consistently met 100 percent of our metrics. You can read more about the metrics on the website.

Engagement Survey, we send out an annual Engagement Survey to all stakeholders and RIR contacts in Q4 of every year and also after every engagement, after every service request, we have a satisfaction survey that’s sent to the requester. And we’ve achieved a 91 percent overall satisfaction rate for those.

Quickly about audits. We’ve had 12 years of exception free audits. We conduct a SOC 2 audit for our TLD network registries. Those include IP allocations, and that was delivered to the NRO back in February.

We conduct a SOC 3 audit for the KSK ceremonies. No exceptions were found within the last year. And that report is actually published on our website. It should be coming out later this month.

Key signing ceremonies. We performed four key signing ceremonies in the past year. We reverted some of our pandemic processes that were in place. So those were put in place to amend in-person contact, such as we signed three quarters instead of every quarter.

And we had staff-only ceremonies but rolled those back. External witnesses have most recently brought in with TCRs and other observers.

We’ve also resumed maintenance that were deferred. We deferred whatever maintenance that was possible to limit in-person contact during the pandemic. And so reintroducing HSM lifecycles, phasing out some of the cards and reassigning new cards, credentials that were in use since we started e-ceremonies.

I’ll talk more about TCR on the next slide, but we’ve started some retuning for our software environment to better support our processes, improve our processes and also to support some of our upcoming efforts.

ICANN has started drafting a design team to study changing of cryptographic algorithm that’s used to sign the DNS Root Zone. That is still in the design process, but this is a pretty involved process.

So once the design team produces a draft plan, this will be opened up for public comment and it will be available on the ICANN website if you want to follow along in the process.

In the short term, we’re looking to generate a new key for the next Trust Anchor rollover in the 2018 round. This is good hygiene and just good practice. It’s still a multiyear process. So we need to publish the new key, have it be used as a standby key, and to allow some time for the community to prepare for the rollover.

Talking a bit more about the TCRs. They’re colloquially known as the key holders of the Internet. They’re basically volunteers that oversee the ICANN and how IANA runs the KSK ceremonies to sign the root servers.

They are drawn from the community and expected to basically be observers and provide feedback and oversight over the process.

They’re usually from security backgrounds. But not necessarily have to be, but usually from a security background.

During the pandemic, some of these TCRs have been in place since we started these ceremonies back in 2010. Some of them have indicated they want to retire.

We’ve kind of placed that on hold during the pandemic. But recently we’ve started retiring them a bit about one per ceremony. And now we’re looking for new TCRs to kind of replace them, and we also want to diversify our pool of TCRs, especially we’ve only had one female TCR to date. And geographically, the Asia Pacific region is also underrepresented. So putting a call out there, if you’re interested, if you know someone that’s interested in becoming a TCR, refer them to our website, and they can submit a statement of interest or feel free to talk to me or email us at [email protected], if you want to find out more about that.

And just ongoing and upcoming work. So IANA is working on developing its RDAP server for IANA-managed resources. This includes the TLDs, .int domains, .arpa domains, reverse zones, reserved IPs and IP allocations. This is intended to augment the IRR RDAP servers.

ICANN has been in discussion with RIRs about Service Level Agreements for reverse DNS. So this was in public comment last year and ICANN is working with RIRs on publishing the final report for that.

And IANA, we have been kind of revamping this and working on public reporting tools in advance of this just to increase transparency on how IANA works with RIRs.

And lastly, in June, engagement begins on the fiscal year ‘25 operating plan and budget. And jumping back to the RDAP servers, we’re shooting for a soft launch middle of this year before we release it to the public. And also plan to open source that as well.

And that’s all. Feel free to ask any questions or contact us if you have any questions.

Hollis Kara: Approach the mics if you have questions for David. Or start typing. Go ahead, R.S.

Robert Seastrom: Hi, I’m Rob Seastrom, I work for Capital One in my day job, and I’m on the Board of Trustees, and I’m one of the TCRs who’s been around since 2010.

And I’d like to emphasize and double down on your request for anyone with a statement of interest, who might be interested to submit a statement of interest. The breadth of possibilities of folks for ICANN to select from is the strength of this group of trusted community representatives.

So if you have even the slightest interest, please step forward and submit a statement of interest. Thank you.

David Dong: Thank you.

Hollis Kara: Thank you. Anything else for David? Here we go.

Scott Johnson: Hi, David. Scott Johnson, SolarNetOne.org.

I’m curious as to the procedure to be followed when a resource managed by IANA grows beyond the scale or requires more specific direction for allocation or management from the IETF than IANA directly provides, such as IP addressing is now managed by the RIRs.

Suppose there were a new resource that were to reach that level. Is there a procedure in place and what is that procedure? Thank you, sir.

David Dong: So, that would depend on the resource itself, I would think. Is there a particular resource you’re thinking of that might be a better…

Scott Johnson: Well, there are some number resources that are associated with recent set of RFCs, 9171 through 9174. These relate to the Bundle Protocol. And I was curious as to the procedures that we would enact in order to create a more robust review and allocation process.

David Dong: I can take that back to the team for some comment, but probably it would have to be more in depth of a topic. I think there’s a formal process in place for that.

Scott Johnson: Absolutely. Thank you, sir.

Hollis Kara: All right. You had to get up.

Kevin Blumberg: Kevin Blumberg, The Wire. There was a wonderful presentation at the recent ICANN meeting where slide decks were shown in terms of the number of protocols that IANA — and maybe this will help the gentleman — the number of different identifiers that IANA handles.

And the synopsis from how I took from it was the numbers is this amount. It’s a very small amount of the total number of different unique identifiers that IANA handles. There’s thousands of them that you handle, correct?

David Dong: Yes, that’s correct.

Hollis Kara: All right. Anything else? I don’t think so. Thank you very much, David.

(Applause.)

In the interest of time, we’re going to shift the Regional Policy Update , that’s the one — it’s been a long day — to tomorrow and go ahead and run our Open Microphone and finish up the day.

We’re going to get John and Bill up here in just a moment. And I will go sit back down.

Open Microphone

Bill Sandiford: Open mic time again. Microphones are open. Feel free to approach the mics with any questions you may have, comments, or get them in online as well for the remote participants.

John Curran: They’re still out there, right? I can’t quite see under the lights. Yes, they’re still there.

Bill Sandiford: Center microphone.

Brian Jones: Brian Jones, Virginia Tech, ARIN AC. Just want to thank ARIN for the opportunity to celebrate 25 years of IPv6 at ARIN 51. Thank you very much.

Bill Sandiford: Thank you.

John Curran: Thank you.

(Applause.)

Bill Sandiford: Anything coming in online?

Hollis Kara: I don’t see anything.

Bill Sandiford: Last call. It’s your chance, Kevin.

(Laughter.)

John Curran: Last call, Kevin.

Hollis Kara: You had to taunt him, Bill.

Kevin Blumberg: You just had to poke the bear.

Kevin Blumberg, The Wire. So, I already know the answer, John. It’s called, there’s this wonderful thing called the ACSP.

I had some interesting chats. We were talking about email and the volume of email yesterday. And one of the things that was brought up was ARIN’s systems are very antiquated.

You’re using Mailman. You’re using old mailing list technology from 25 years ago, and many companies have gone beyond that and allowed you to select interests and have a completely newer way of interfacing with their customers.

And, yes, I could submit it as an ACSP, but my basic suggestion is maybe something that is both regulatory compliant as well as feature-rich for us as customers would be more appropriate than the antiquated Mailing List.

John Curran: Okay, thank you.

Bill Sandiford: Thank you. All right. Going once. Going twice. Nothing online. All right. Thank you, everybody.

(Applause.)

Closing Announcements and Adjournment

Hollis Kara: Just a couple quick things before everybody runs away. Thank you for being here. Man, you guys are ready to get outside. I understand.

Before you go, could we please have a round of applause for our sponsors, Charter, IPv4.Global by Hilco Streambank, and Google.

(Applause.)

Reminder, we’ll be back again tomorrow to finish up. We do have a lot of great reports, so we hope you’ll be back. Breakfast is at 8 a.m., and the meeting will start at 9:00. And have a lovely evening. Thank you so much.

(Applause.)

(Meeting adjourned at 4:30 PM)