Blog
Welcome to the Hardenize blog. This is where we will document our journey as we make the Internet a more secure place and have some fun and excitement along the way.
When it comes to network infrastructure, both security and availability begin with domain names. You can have the best security elsewhere, but if your domain name is taken away from you, it all falls apart. With today’s release, we’re adding domain name registration monitoring to our capabilities.
Domain monitoring is a feature we’ve been working on for a long while. Today we deployed with support for both domain name lookup protocols—the older WHOIS, and the newer RDAP. We monitor our customers’ domain names on a daily basis, using both protocols to acquire as much data as we can.
Overall, it’s been a lot of work to get to this point, and we still have ways in which we can improve. Being able to reliably obtain domain name information is not easy. The chief obstacle is that today, still, WHOIS remains the main domain lookup protocol, but obtaining the data and interpreting it isn’t at all easy, owing to the protocol’s simplicity and lack of structure. RDAP is a better solution, but it’s not as widely supported today. On top of all this, anyone wanting to perform domain lookup for legitimate reasons has to fight for access with armies of spammers consuming WHOIS (and possibly) RDAP to accumulate as much data as possible.
WHOIS is one of the oldest internet protocols and it’s devastatingly simple. It’s most recently documented in RFC 3912. You connect to a server on port 43, send your query, which is just a line of text, and get a longer multiline response back. Because the query and the response formats are not standardized in any way, there are multiple challenges related to communicating what is wanted, knowing what character set is used for the response, interpreting whether the response contains data or an error message, and finally, converting the data into fields with meaning.
For illustration, let’s look at one response for twitter.com:
Domain Name: TWITTER.COM Registry Domain ID: 18195971_DOMAIN_COM-VRSN Registrar WHOIS Server: whois.corporatedomains.com Registrar URL: http://cscdbs.com Updated Date: 2022-01-17T06:08:20Z Creation Date: 2000-01-21T16:28:17Z Registry Expiry Date: 2023-01-21T16:28:17Z Registrar: CSC Corporate Domains, Inc. Registrar IANA ID: 299 Registrar Abuse Contact Email: domainabuse@cscglobal.com Registrar Abuse Contact Phone: 8887802723 Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited Domain Status: serverDeleteProhibited https://icann.org/epp#serverDeleteProhibited Domain Status: serverTransferProhibited https://icann.org/epp#serverTransferProhibited Domain Status: serverUpdateProhibited https://icann.org/epp#serverUpdateProhibited Name Server: A.R06.TWTRDNS.NET Name Server: B.R06.TWTRDNS.NET Name Server: C.R06.TWTRDNS.NET Name Server: D.R06.TWTRDNS.NET Name Server: D01-01.NS.TWTRDNS.NET Name Server: D01-02.NS.TWTRDNS.NET Name Server: NS3.P34.DYNECT.NET Name Server: NS4.P34.DYNECT.NET DNSSEC: unsigned URL of the ICANN Whois Inaccuracy Complaint Form: https://www.icann.org/wicf/ >>> Last update of whois database: 2022-03-29T13:41:12Z <<<
All right—you might say—that‘s not too bad. And it wouldn‘t be, if this was the only format you had to process, but then you have to take into account that there are thousands of TLDs and even many more registrars, all using potentially different formats, all possibly in a state of flux.
Other problems include knowing which WHOIS server to talk to, which is not always obvious. For some domains, WHOIS isn‘t available or it‘s provided via a web form.
RDAP, which stands for Registration Data Access Protocol, is the protocol designed to replace WHOIS after the problems we mentioned (and some more) became obvious. The IETF looked at how WHOIS is used and designed a new protocol around the existing data. It consists of several RFCs and so the best way to understand it is to look at the documents produced by the IETF‘s regex working group.
RDAP is a modern protocol built around HTTP and JSON. It‘s a breeze to process (albeit still complicated) and a great improvement over WHOIS. Unfortunately, RDAP isn‘t yet universally supported. ICANN has required RDAP support for TLDs under its jurisdiction since 2019, but nothing pushed adoption among ccTLDs, where support is lacking.
In the end, we do what we can with what we have. This blog post only scratches the surface of the problem domain. We didn‘t talk how, in many instances, the domain lookup responses don‘t contain some vital information, for example status of registrar and registry locks. Or that WHOIS and RDAP protocols are used for lookup of objects other than domain names, for example IP addresses. There‘s much more to say, perhaps in our future blog posts.
Although in the long term RDAP may be the answer, WHOIS is here to stay. Our goal is to obtain reliable and actionable data, and we do it by combining what we get from both WHOIS and RDAP. And we continue to improve our coverage and understanding of the data, even if that means, in the future, resorting to performing domain name lookups using web forms whilst solving captchas.