P99 0 ms* autocomplete for 240M domain names

(ruurtjan.com)

220 points | by dbalatero 1 day ago

19 comments

  • skybrian 1 day ago
    This autocomplete suggests domains that don't exist. You can just type garbage and it will suggest something, but then if you go there, there are no records.

    It seems like one purpose of an autocomplete box is help you avoid typos, so that makes it less useful.

    • pul 1 day ago
      Author here.

      Yes, it will autocomplete the most popular domains first, then known registered domains, and then fall back to appending popular TLDs.

      The reason is that it's impossible to enumerate all domains. gTLDs (like .com, .info) mostly publish their zone files, but ccTLDs (like .cn, .de) usually don't.

      But maybe you're right. After reading these comments, it's clearly confusing to people.

      • 8organicbits 1 day ago
        There are limits to how accurate you can make this. Zones like .xyz renew at 18%, so you'd expect 0.2% of those domains to expire without renewal each day. The tranco list is based on a 30 day look-back and I've seen a small percent of those domains lack name servers, even for the latest list.

        Similarly, domains can be registered since the last time you downloaded your lists. So you never have a complete or accurate list.

        I think it's perfectly reasonable to suggest names that have recently expired or domains that could have been recently registered. The alternative requires an NS lookup.

      • andai 1 day ago
        I'm not sure I understand the reasoning behind appending the TLDs?

        Can't you just append any TLD to anything and get a domain that "might" exist? Or am I missing something?

    • pierrefermat1 1 day ago
      Yes, OP seems to have completely lost touch with what is actually useful vs optimizing metrics for the sake of it.
      • lnenad 1 day ago
        But there is a cool blog post about it though.
    • bawolff 1 day ago
      The autocomplete seemed to suggest real domains to me. If you typed in garbage it suggested the garbage plus a bunch of common TLDs - that seems like a reasonable choice to me. In any case, if your issue is the list of domains to suggest, its trivial to change that to a different list.
    • addag 1 day ago
      This behavior seems to happen only when there is no more words in the list that can complete the current string. Otherwise, it shows only existing domains.

      I guess this behavior helps to autocomplete the domain extension for less-known domains that are not in the search list.

      • Symbiote 1 day ago
        It doesn't work correctly with third level domains.

        speedtest.me.uk exists, but typing speedtest.me does not suggest it, and speedtest.me.u suggests speedtest.me.us first (doesn't exist).

        (.me.uk is a second level domain category in .uk, intended for personal domains.)

    • 1718627440 1 day ago
      That's also the old behaviour of browsers and the current one of text browsers. I fail to see how this is wrong.

          $ lynx doesnotexist2
      
          Looking up doesnotexist2 first
          Looking up www.doesnotexist2.com, guessing...
          Looking up www.doesnotexist2.edu, guessing...
          Looking up www.doesnotexist2.net, guessing...
          Looking up www.doesnotexist2.org, guessing...
          Can't Access `file://localhost/home/user/doesnotexist2'
      • 1vuio0pswjnm7 21 hours ago
        "That's also the old behaviour of browsers and the current one of text browsers."

        As a daily text-only browser user for over three decades, using it as a "daily driver", I'm no doubt biased. I was a lynx user before links was released. I would never go back to using lynx. When I see HN comments suggesting lynx as an example of text-only browsing I question whether the commenter is actually using it as "daily driver", or looked at the source code and compared it with alternatives

           links doesnotexist2
        
        does not trigger a DNS lookup

        Unlike elinks, w3m, etc.

        One of the many reasons I prefer links (no graphics)

        I use links as an offline HTML reader; I am opening local HTML files every day

        That said, it's relatively easy to modify these programs to behave according to personal preference; they are relatively quick and easy to compile

        One could remove the unwanted DNS lookups, for example

        Contrast this with the so-called "modern" browser that almost zero users have the time and resources to modify and compile themselves

        Instead, we see dissatisfied browser users beg browser vendors to make changes

        To no avail

        There are other ways to prevent unwanted DNS lookups besides modifying the source code

        For example, I run own authoritative DNS on the loopback, with custom root.zone

        All lookups by default return the same address: the address of the local forward proxy

        The proxy's memory contains DNS data precollected periodically in bulk from a variety of sources. At startup/restart, the proxy loads the data into memory from a plain text file (a "map")

        The map would not contain any DNS data for "doesnotexist2"

        There is no way for DNS lookups to leave the loopback nor any way for HTTP requests to be sent to destinations other than the ones I approve

        No Javascript engine in text-only browser

        For use with "modern" (read: uncontrollable) browsers the proxy can deny requests for URLs ending in .js, add a Content-Security Policy response header that disables Javascript and, if the CSP were to be ignored, can rewrite pages to comment out or delete inline JS

        • 1vuio0pswjnm7 1 hour ago
          Last sentence is sort of convoluted

          What I mean is if "modern" browser vendors change their popular browsers to make it more difficult for users to control others' Javascript, we can use URL blocking, URL rewriting and/or page rewriting to modify or delete inline or sourced Javascript and/or substitute the user's own Javascripts

      • Retr0id 1 day ago
        I hate this behaviour because it means if I typo an internal hostname suddenly it gets broadcast to everyone's DNS resolver, and wastes some time in the process (and in the worst case, takes me to a random website I never intended to visit). I always have to rummage around in about:config or whatever to disable it, because that behaviour is never what I expect nor intend.
  • chrismorgan 1 day ago
    Using keyup makes no sense and is inconsistent with user expectations. For triggering actions (which includes normal typing), you only ever use keydown. (Well, there’s one exception for reasons unclear to me: activating a button by pressing Space. That triggers on keyup like how clicks are on release, while Enter triggers on keydown.) Keyup is limited to things where you’re constantly reacting to the state of a key, as is common in games.

    This affects the functionality, too. It is in fact introducing latency by using keyup instead of keydown. Feels bad.

    • danbruc 1 day ago
      For triggering actions (which includes normal typing), you only ever use keydown.

      I think you have this wrong, actions generally occur on button release, until that you can move the mouse cursor to a different target, tab to another control, use [ESC] to cancel, and so on. Typing, moving a slider with the cursor keys, and similar things that make use of key repetition while holding down the key are the exception to this.

      • chrismorgan 1 day ago
        I am speaking specifically of keyboards. For mice, it’s release. (When I made a similar comment two months ago on Lobsters when this article came up there, I mentioned this, but decided to leave it out this time, thinking the context was clear enough. I think I was wrong.)
        • okamiueru 1 day ago
          If it matters, it seemed pretty clear to me. You're also explicit about it, so I'm not sure how one would think you were referring to mouse input. Unless people type by clicking on a virtual keyboard? Benefit of the doubt can only stretch so far.
      • okamiueru 1 day ago
        Can you give some specific examples? In my experience (which I also verified to make sure), all of the things you mention act on keydown. The exceptions are on multi-key combinations, which follow a special key that is held down while modifiers are entered, where the action is completed by releasing this special key (ALT + TAB [...TAB | SHIFT]) being a common example.

        TAB is otherwise immediate, as is ESC, and the examples of typing, as well. So I'm left wondering, how do they have it wrong?

        • invalidusernam3 1 day ago
          Non latin characters are sometimes inputted by holding down a key. For example on my keyboard holding down z shows ź,ž and ż with a corresponding number to select which one I would like to type. I haven't checked but one would assume if you're detecting keydown the input at that time will be z, and keyup would record ž (or whatever was selected).
          • Xirdus 1 day ago
            Physical keyboards and virtual keyboards are different. Traditionally, ź and ż (my keyboard doesn't have ž) are inputted on keydown X or keydown Z while Alt modifier is active.
        • wongarsu 1 day ago
          > all of the things you mention act on keydown

          Because they listed the exceptions to the "act on keyup" rule

          • chrismorgan 1 day ago
            If only one case follows your rule, and everything else ever implemented from the start of personal computers is an exception, your rule is awry.
        • stavros 1 day ago
          The GP means that, if you hold space down on a button, you can then tab to another button without activating the first one.
    • hnlmorg 1 day ago
      Key down is used for events that can be duplicated by holding the key. Eg in a text editor, when you want multiples of the same letters, you’d press and hold the key.

      Key up is used for when it’s important to only have one occurrence of that event.

      Technically you could write code that made key down only react once. But the logical separation makes some sense.

      • chrismorgan 1 day ago
        Give me an example. Any example beyond this Space one we’re discussing; because I don’t know of any.
        • hnlmorg 22 hours ago
          Examples where key down is used as an input event?

          Basically anything that requires text input from OS-native text input fields to IDEs, word processors, spreadsheets, web browsers, terminal emulators and so on and so forth.

          • chrismorgan 22 hours ago
            Key UP!

            (I confess I’m getting frustrated at the ridiculously large fraction of messages in this thread that have somehow completely misunderstood things.)

            • hnlmorg 19 hours ago
              Your message was ambiguous to me. But I will admit that I was also quite hungover so it’s possible that the error was mine.

              In answer to your question: I’ve only really used keyup for either one time events (like keyboard shortcuts for UI elements) and for ending the status of repeated events (eg movement in games, dragging UI elements, etc).

              I think if you were to search for key up events in GitHub, most of references might likely be games and other visual interactions rather than text entry.

        • youngtaff 1 day ago
          I can see both viewpoints here and don’t have a strong personal view

          But for reference browsers fire the keypress event on keyup

    • zxexz 1 day ago
      As a user, I’ve not thought too much about this before now. I agree with you mostly, but the keyup on space behaviour actually feels so innate I’d hate any change to it. Keydown on space is “jump”; nothing else.
      • chrismorgan 1 day ago
        I’d just like to hear an explanation of why it is. Because if you handled “is key currently pressed” events with polling rather than events (which is how people almost always consume them logically), it would be the only thing in the entire world that I can think of that would/should use keyup.

        Edit: I think people are misunderstanding me. I’m asking for an explanation of why, when you have a button focused, pressing Space activates on key up, rather than key down like literally everything else on the keyboard, including Enter on a focused button. (Mouse activates on release.)

        • Rygian 1 day ago
          Because in those contexts, the Space key is being used as a physical mirror of the on-screen button, and on-screen buttons are UX'd to perform action when released.

          These two interactions should behave identically:

          - Hover over a button, press mouse button 1, release button 1.

          - Tab over to a button, press spacebar, release spacebar.

          Why on-screen buttons are UX'd to behave this way is a logical follow-up question. I'd wager that it gives a means to bail out of the clicking (e.g. by moving the mouse out of the on-screen button before releasing mousebutton 1, or by pressing Esc while having spacebar still depressed).

          • regularfry 1 day ago
            Yes, the bailing-out explanation is right. With a mouse it's intentional that you should be able to push the pointer off the button with the mouse button still pressed to cancel.
          • chrismorgan 1 day ago
            (Esc while Space is held doesn’t cancel, but Tab does.)

            You’re only answering half of the inconsistency: because Enter activates buttons like any other key, on keydown.

            • Rygian 1 day ago
              While spacebar is depressed:

              Qt: Esc cancels. Tab cancels.

              Gtk-4: Spacebar auto-clicks before keyup (probably due to auto-repeat).

              Win32: Esc sends Esc to the window. Tab cancels.

              Firefox: Esc does nothing. Tab cancels.

              So, mileage does vary.

              Re. Enter key, that's a different key, so different discussion.

              • chrismorgan 23 hours ago
                Under Linux, Firefox, Chromium and GNOME Web (which is WebKit-based; a.k.a. Epiphany) all have Tab cancel but not Esc.

                The GTK 4 behaviour feels very much like a bug. I can reproduce it in gtk4-demo and in gtk3-demo. It leads to obviously-nonsensical behaviour, including messed up window focus. I’m tempted to file a bug report, but haven’t had such productive experiences with filing things on GNOME historically as with Firefox or even Chromium.

                Why should Enter being a different key be a different discussion? They’re both used to activate buttons, and one is behaving like every single other keyboard interaction, while the other isn’t. (Related: if a link is focused, Enter activates it, Space doesn’t, just doing the normal scroll-down-one-page.)

                • Rygian 8 hours ago
                  > Why should Enter being a different key be a different discussion?

                  > if a link is focused, Enter activates it, Space doesn’t

                  It seems that you answered your own question: different keys, different purposes.

                  Enter is for "execute the focused action". Spacebar is for "operate the button".

            • b112 1 day ago
              Just diving in without full context, but, enter is often a completion action. Historically often a full carriage return. It often means "submit data".

              So this could be the oddity

        • podocarp 1 day ago
          It just matches real life. What about lmb to shoot? You don't fire a gun on trigger release, you fire it on trigger down. Walking? Would be weird moving only on keyup.
          • wky 1 day ago
            This raises the question of why real life buttons and virtual buttons behave differently. My unsubstantiated guess is that clicks act on release to give the opportunity to slide off the button to abort, and/or because the button would disappear while pressing if it ex. submits a form.
        • dangond 1 day ago
          Jump is keydown because many games let you adjust your jump height based on how long you hold down space after the jump starts.
          • flaunf221 1 day ago
            Games use keydown because it is more responsive and control responsiveness is vital to dynamic gameplay feeling good. There are some exceptions - if action can be charged, then it is on keydown. And then on gamepad sometimes actions are on release when developer has too many actions and not enough buttons, so it can be keyup vs hold.
        • stavros 1 day ago
          I may be wrong on this, but I think that pressing esc while space is held down cancels the button press, like moving the mouse off the key while holding down the click does. I'd need to test this, though.
    • vrighter 5 hours ago
      this is backwards. Go on.... try it anywhere. Just the start menu (if you're on windows) is enough.
    • dawnerd 1 day ago
      That’s where the input event is really useful as it allows for more input options than just keyboard and then you don’t have to deal with figuring out if it’s down or up.
    • 1718627440 1 day ago
      On my own computer I can open a context menu and select an action with a single click. Whenever I need to use MS Windows, I always need two and I find that infuriating every time.
  • ViscountPenguin 1 day ago
    Unfortunately this approach doesn't feel that great down here in Australia, definitely a function of latency.

    I think you could get a lot closer by framing this as an optimization problem, where you use the full alphabet dictionary, but add a residual prediction which aims to cover as much of the remaining domain name tree as possible weighted by popularity. This tree could then be pre-baked and stored with the same system. This would probably get you p99 0ms even in Australia.

    • pul 1 day ago
      Interesting, I didn't think of that yet.

      I think most people search for domains they own, which doesn't correlate with Tranco popularity ranking. Treating 'popularity' not as a function of visitors, but as a function of number of known domain names with that prefix could work, though.

      • ViscountPenguin 15 hours ago
        Fair enough, if people visit frequently you could probably save their previous searches as a cookie too.
  • oersted 1 day ago
    Why not just trigger the fetch on keyDown and show it as soon as the response arrives, as usual?

    The time it takes to press a key is a reasonable target to aim at for API latency I suppose, but it is still an arbitrary target. Waiting to display until keyUp just adds more latency if your API is faster. Having it synced with keyUp doesn't make it feel more immediate to me.

  • kevmo314 1 day ago
    If you’d like to reduce the network latency further you can store each trie node as a file, naming it conveniently the prefix path to that node. Then dump the few hundred million files onto R2.

    Now the traversal can be done completely via CDN lookups!

    • pul 1 day ago
      Cloudflare R2 doesn't distribute files globally. They only cache requested files in PoPs. So then it would function the same as an API behind the Cloudflare CDN.
      • kevmo314 1 day ago
        It would skip the Cloudflare to server network hop that the author mentions being rather expensive for cache misses.
        • pul 1 day ago
          It would replace the Cloudflare to server network hop with a cloudflare PoP to R2 hop. That covers approximately the same geographical distance and results in similar latency.

          (I'm the author btw)

  • phil294 1 day ago
    Tangentially, I built something similar a few years back, at link-archive.org: https://web.archive.org/web/20220127233707/https://link-arch...

    3B existing URLs extracted from CommonCrawl, with instant search results. It was a fun project but didn't serve much real-world purpose besides curiosity and discovery. So I eventually ditched it, primarily because the link DB was a whopping 500 GiB in size, too much to just keep hosting.

    I just used SQLite FTS5 as the backend search engine. Just a few lines of code, but immediate response from a 0.5 TiB DB. SQLite is amazing.

  • camel_gopher 1 day ago
    Clever but that’s not how we measure latency.
    • dbalatero 1 day ago
      When it comes to UX, perceived latency is king.
      • wky 1 day ago
        The perceived latency starts from keydown, not keyup. Redefining latency to start at keyup reduces measured latency, not perceived latency, and delaying the visual display to keyup makes perceived latency strictly worse, not better. Even sticking with the keyup definition, just displaying the result as soon as it is available gives the possibility of negative (defined) latency.
        • weird-eye-issue 1 day ago
          I think you don't fully understand. They aren't just doing a search based on the key being pressed down. They already did a search based on the previous characters that returned results for all possible next characters. So by the time you type a second character it just checks results locally from the search that had likely already been returned from when you had typed the previous character.
          • wky 1 day ago
            They do that, yet proceed to kneecap the perceived latency by delaying the render.

            > And on keyUp (the user releases the key), we render the suggestions.

            • weird-eye-issue 1 day ago
              There are a large number of edge cases where rendering it on key down would not be desirable (user types Ctrl+v, how accented characters are enetered, how some Asian languages get handled, backspace/delete, user holds a key down, etc). This is getting into the point of optimization where it simply does not matter and can actually harm things
              • wky 1 day ago
                Holding a key would indeed require both on key down and up. I'd still argue that handling rerender on key down (or to fully match whatever OS/browser semantics, oninput) makes more sense than key up. Pressing Ctrl+V for example produces feedback as soon as "V" is pressed, not when Ctrl and/or V is released.

                Interestingly Ctrl+V on OP's website does update immediately when Ctrl+V is pressed.

                • weird-eye-issue 1 day ago
                  Why would you argue that makes more sense? That is not how any autocomplete works that I know of, it's not like this is a new pattern.
                  • Dylan16807 16 hours ago
                    What autocompletes are you testing? I set my repeat delay to be very long so it doesn't interfere with the test, and then went to google and held down some keys for half a second. It types the letter and autocompletes long before I let go.

                    Every UI I can find types the letter on keydown, and I don't know why you'd want typing and autocomplete to use different timings.

                    Waiting for release is for mice and touch, not keyboards. And that's only because they have to deal with dragging.

                    • weird-eye-issue 16 hours ago
                      Your test is flawed, it's triggering keyup if it's showing in the input.
                      • Dylan16807 15 hours ago
                        I go to the text box in HN and hold down a key.

                        The letter appears instantly, then after a moment a lot more copies of the same letter start showing up rapid-fire.

                        What are you calling flawed about my test?

                        This happens on both Windows 10 and Linux Mint, in Firefox.

                        • weird-eye-issue 15 hours ago
                          Overall the correct event to listen to would be the input even not the keydown event since it handles the edge cases I mentioned earlier

                          I never meant to infer that the keyup event is specifically what should be listened to

      • vasco 1 day ago
        What you said might be true and yet what OPs AI agent did wasn't measure latency of the service nor calculated a p99 for it.
  • piterrro 1 day ago
    I just typed a random sequence of the characters, long enough to be certain such domain doesnt exist. No only, the browser send an autocomplete request for every keystroke but for each request it returned a set of proposed domain names (which I'm 100% certain doesnt exist). At this point, how do I understand which results are legit and which are fake? Also, it would be nice to highlight the typed part in the result set so I can visually see what matches exactly.
    • addag 1 day ago
      For the nonexisting domains, it seems like it only autocomplete with the possible domain extensions, (e.g .com, .org, ....) as the search list is non-exhaustive. But it could indeed be improved by not sending autocomplete requests anymore.
    • pul 1 day ago
      You're not the first to mention this, so clearly users expect something different from what I've designed it for. I think I'll drop the TLD postfix suggestions.
  • bagels 1 day ago
    Looks more like 500ms?
    • konsalexee 1 day ago
      Where did you get that?? Did you even read the post?
  • andai 1 day ago
    >I've designed the API to first search Tranco (the head), and then CZDS (the tail) if necessary.

    How do you know if it's necessary? If Tranco returns less than eight?

    • whalesalad 1 day ago
      Tranco is only top 1M domains.
  • cortesoft 1 day ago
    KeyDown events don’t work great for mobile, though.
  • pupppet 1 day ago
    Autocomplete aside, this is a pretty nifty tool.
  • ape4 1 day ago
    How are the icons (for the domains) fetched so quickly?
  • pixelpoet 1 day ago
    Pretty sure we mean < 1ms rather than actually instantaneous.
    • weird-eye-issue 1 day ago
      On a technical level yes but once you factor in monitor refresh rates then you can get to levels of optimization where it simply doesn't matter because you are constrained by waiting for the refresh rate anyways.
    • lovich 1 day ago
      Its like tic tacs saying they are 0 calories because they got the per serving size down low enough to round to 0.
  • ChannelFence 1 day ago
    its pretty clever but what happns when someone pastes a domain or uses IME or voice input? the api being that fast is still impressive.
    • ltrg 1 day ago
      Or uses swipe gestures on mobile keyboard!
  • Sarthakofficial 1 day ago
    [flagged]
  • AboozarEsmaili 1 day ago
    [flagged]
  • K0IN 1 day ago
    [dead]