Seven months of watching people typo NLTK on PyPI

I was cleaning out my NAS recently and found some data from when I was typosquatting NLTK on PyPI, so I figured I’d write up something short and share.

The story goes: several years ago I was presenting a demo and attempted to install NLTK - the Natural Language Toolkit - with pip install ntlk. Three times in a row, I kept getting:

$ pip install ntlk
Defaulting to user installation because normal site-packages is not writeable
ERROR: Could not find a version that satisfies the requirement ntlk (from versions: none)
ERROR: No matching distribution found for ntlk

After retrying and confirming that PyPI was online, that I was using a compatible Python version, that I hadn’t lost my mind, etc., I eventually noticed I’d typed the letters in the wrong order. Me, a security guy, making exactly the kind of two-key typo that opens the door to malware running on my system! :(

Chatting at work later, even when saying the package name out loud, I kept confusing N-T-L-K with N-L-T-K again. Something about the spelling ntlk sounded weirdly right to me, even when I knew it was wrong.¹ I got curious and searched StackOverflow at the time, where I saw ntlk had appeared 244 times - just over 1% of the results for nltk which appeared 21,228 times - a pretty common typo.

I wondered how many typos that would be on the command line. At the time, nltk was in the top 250 Python packages and is downloaded two million times per week (and today, nltk is downloaded over 15 million times per week) - while the vast majority are automated and not able to be typosquatted, ex. via pip install -r requirements.txt, surely it wouldn’t be zero.

Let’s Find Out

So, I made a benign typosquatting package: I registered ntlk on PyPI and uploaded a package that tells you that you made a typo and should run pip install nltk instead.

This was very simple to create. When you run pip install <whatever>, pip downloads the package sdist and invokes setup.py. So you can easily write a package with a custom install command class to print a message or whatever you want, and others such as Python Lulz Authority have done this in the past for requirements.txt. I wrote a very simple setup which would fire an HTTPS POST to a domain I set up - allowing me to track the issue and see how many real installs were happening, rather than just downloads² - before raising a SystemExit with a descriptive message, ex.

from setuptools import setup
from setuptools.command.install import install

class AbortInstall(install):
    def run(self):
        url = "<where to send a short report>"
        try:
            # Python3-compatible urllib setup
            from urllib.request import Request, urlopen

            req = Request(url, method="POST")
        except ImportError:
            # Python2-compatible urllib setup
            from urllib2 import Request, urlopen

            req = Request(url, data="python2=bad")

        try:
            # Note that no system data is collected
            req.add_header("Package", "ntlk")

            # Here, the report is sent, but we don't care about the read data
            urlopen(req).read()
        except Exception:
            # If an error happens or this is not possible, OK, whatever
            pass

        # And a message explaining the issue is raised
        raise SystemExit("<Your descriptive message!>")

if __name__ == "__main__":
    setup(cmdclass={"install": AbortInstall})

After uploading, anyone making the mistake I did would now see a helpful message when they beefed an install:

$ pip install ntlk
Collecting ntlk
Building wheels for collected packages: ntlk
  Building wheel for ntlk (setup.py) ... error
  Complete output (13 lines):
  running bdist_wheel
  running build
  installing to build/bdist.linux-x86_64/wheel
  running install

  You probably meant to install NLTK (the Natural Language Toolkit).
  You have attempted to install NTLK instead. This is an empty package
  to help prevent typosquatting. To install NLTK, try this instead:
  pip install nltk

  Stay safe,
  tweedge <3

And once this was uploaded, all I needed to do was wait.

The Results

The package was available for about seven months³ before PyPI’s maintainers pulled it, as typosquatted packages aren’t permitted on PyPI even if benign.⁴ In that time, it was installed a total of 28,839 times by 10,938 unique IPs - roughly 135 installations per day.

Unfortunately, while I used to have more detailed files which would show ex. when the installs were logged (to see whether or not these happened during business hours, of late at night when folks are groggy, etc.), the only thing kicking around in my NAS was summary information about the number of installs and the source IPs, ASN, and rough geolocation for each. That can still tell us a lot of interesting things, with the source ASN for attempted installations of ntlk being the most interesting by far:

Installs by ASN

Breaking down the 28,839 installs by the type of organization behind each ASN:

The residential/ISP majority is what you’d expect: real people sitting at terminals who made a mistake. This is what the population I’d expected to see and was excited to get confirmation from. The more interesting story is in the tail.

Cloud / Hosting (17.9%) is dominated by Google (2,608 requests across 808 IPs), Microsoft Azure (851), Amazon AWS (855 combined across two ASNs), Alibaba Cloud (143), and DigitalOcean (75). I think most of these are likely hosts that folks mistyped an install on when remoted into, and unfortunately we can’t tell what company was renting that particular system to hypothesize about what it’d have access to. That said, some could be security systems that probe PyPI packages - these shouldn’t be taken as 100% real typos.

Academic (4.0%) is one of the more interesting segments. A third of it (411 requests) comes from Chinese academic networks - CERNET and CERNET2 combined - which makes sense given NLTK’s use in computational linguistics coursework. Beyond China, JANET (UK’s national research network, 67 requests), RENATER (France, 59), and SWITCH (Switzerland, 44) all show up. Specific universities such as Peking University (26 requests) and Shanghai Jiao Tong University (25 requests) suggest students working through NLP assignments hitting the same typo.

Enterprise (0.2%) is small but fascinating. Cisco’s enterprise network (ASN109) contributed 20 requests, as did Apple Engineering (9 requests from ASN714), American Express (6), Roche’s global corporate IP network (6), SAP (5), Pfizer (2), Applied Materials (2). These are all corporate egress IPs, likely meaning someone in an internal dev environment had ntlk in a requirements file or typed it at a terminal.

Government (0.1%) is what I’m most interested in, personally: what appear to be genuine government- or government-adjacent ASNs account for 39 requests from 22 unique IPs. That includes the US FDA (2 requests from ASN22828), Queensland Government Business IT in Australia (2), and Singapore’s DSO National Laboratories - the country’s primary defense research agency (2).

The reason this matters is obvious: if ntlk had been malicious instead of a blinking “you made a typo, here’s the fixed command” sign, someone running a Python environment anywhere from academia to major businesses to defense research agencies could have had malware silently installed on their system. This is why many attackers run typosquatting campaigns when they see an opportunity - the install phase of pip install runs arbitrary Python before you’ve had any chance to review or approve it. The blast radius of a malicious ntlk would have been wide if it was well-disguised, and incident responders across the globe could have had a bad month if it sat undetected.

Installs by Region

The original goal that I had was that I would love to figure out if there are specific factors leading to higher rates of typos which could increase or decrease the effect of these attacks in the future - like whether typos are more likely to happen during the day (in professional or academic environments) vs. overnight (when people are sleepy), or linguistic factors like common letter sequences in users’ primary languages, etc. Unfortunately, I don’t have the exact times that each request came in anymore, so we can’t do a time-of-day analysis. But we could look at the region-level data and see if that points us towards any areas of further research.

For example, you can look at ntlk installs by country (based on IP geolocation, top 10):

To see if any region appears to be installing ntlk more or less frequently, which could point to further areas of future study, we can use NLTK usage within each geographic region as a baseline (using ClickPy data). However, it turns out that NLTK downloads only roughly correlate to ntlk installs.

The US being first in NLTK downloads isn’t surprising given that it led the pack in ntlk installs. But rather than being a 22% share, fully 85.8% of legitimate NLTK downloads happened within the US. The data being very US-centric is likely because the US is a cloud hub and between automated pipelines, mirrors, and build systems, many install from requirements.txt or dependencies of other installations and therefore 1. install NLTK much more often than people do and 2. aren’t subject to installing ntlk via typo.

Still, let’s allow ourselves to go off the rails a little, and into less scientific territory. If we exclude the US from both datasets and compare the remaining shares as a ratio (ntlk% / NLTK%), you can find countries which are:

• Over-represented in ntlk vs. NLTK,⁵ countries where humans make this typo at a higher rate than their NLTK download share would predict

• Under-represented in ntlk vs. NLTK, countries where humans make this typo at a lower rate than their NLTK download share would predict

Unfortunately, we just don’t have granular enough data to make a firm analysis here. My data only says typos are universal. This typo was being made by a lot of people around the world, and as such ntlk ended up in a lot of interesting places and surely threw warnings to lots of very interesting people. Sadly, there’s nothing more we can tell for sure about why typos were more- or less-likely to happen by geography at this level, and future work into typosquatting should likely be looking at more specific occurrences with deeper data collected on the environment they’re occurring in.

Conclusion

This was a fun project to learn a little more about Python packaging with, respond to the occasional “lol just installed ntlk thanks for the heads up,” and produce interesting data to root through. Unfortunately, there’s a lot more to do which will help harden systems against typosquatting - especially now with AI agents searching for and installing packages on systems autonomously - we should be looking to making sandboxing easier for everyone and tightening software supply chain controls in corporate environments.

For those curious, the ntlk package is available on GitHub and I’ve made a zip of its final state here.⁶ The code is very short and easy to understand. If you’d like to dig into the data yourself, I’ve also published anonymized install data - all 28,839 records with the source country, timezone, ASN, Python version, package name, and package version for each install. However, IPs and IP geolocation data have been removed for privacy.

There is nothing next for ntlk - now that it’s been removed, the package name on PyPI is denylisted, and nobody else can upload a new version of ntlk. This is much more durable than a benign placeholder which could be misused in an account takeover attack or similar. So while I’m sad the experience has now reverted to:

$ pip install ntlk
Defaulting to user installation because normal site-packages is not writeable
ERROR: Could not find a version that satisfies the requirement ntlk (from versions: none)
ERROR: No matching distribution found for ntlk

From a security perspective, it’s the right call - so future typosquats aren’t being prevented by some random account holding a package and pinkie-promising it’ll only display a helpful warning message forever. As always, I moved off to another project - more to come.

Appendix