Password Cracking: Pairing Down Wordlists via the Linux Command Line
In a couple of previous articles (here and here), we learned several things to consider regarding the collection and combination of password wordlists, and how to use the cat command to combine multiple wordlists in to one. You may have concatenated a bunch of wordlists together to create one giant wordlist for long, unattended cracking sessions, but there may be some problems when doing that.
When cracking a password, each and every password that’s in a wordlist takes time to hash, compare against the password hash, and then either be successful or (most likely) throw away and continue on. Because of this, we want to keep wordlists as concise as possible. When concatenating many wordlists together, you’ll run in to two issues specifically, and those are duplicate entries, and password lengths based on known rules or policies.
Modifying Wordlist Lengths with the ‘grep’ Command
The first thing that we may want to do to drastically reduce the time it takes to crack passwords is to cut down a wordlist based on the requirements of the password that we’re attempting to crack. For example, you may have a giant, multi-gigabyte password list that days hours and hours to run. It may contain passwords anywhere from 1 characters all the way up to 20+ characters. That’s fine, but what if we have a list of hashes and we know that their network policy requires that the passwords be a minimum of 8 characters in length. We don’t want to bother trying the 1-7 character passwords, because that’s a waste of time.
Another scenario – say you’re cracking WPA2 pre-shared key passwords. The requirements for a WPA2-PSK password is that it be at least 8 characters, but no more than 63 characters. Why would we bother with <8 character passwords or >63 character passwords? Again, waste of time.
The following command uses WPA2-PSK as an example for length, and cuts all lines out of a password wordlist file that don’t meet the requirements for a WPA2-PSK password. This syntax uses a megawordlist file for example, and outputs a new file which has only passwords that range in length from 8 to 63 characters, which comply with WPA2-PSK requirements.
grep -x '.\{8,63\}' megawordlist > megawordlist_wpa2
We’re left with a file called megawordslist_wpa2 that we can use to crack WPA2-PSK hashes with no wasted time.
Removing Duplicate Entries from a Wordlist
A lot of wordlists have very common passwords that will also appear on other wordlists. When combining multiple lists, this can result in a lot of wasted time. You can easily concatenate multiple text files together with the following command, which is great for creating a large wordlist from many small wordlists.
cat wordlist1 wordlist2 wordlist3 > megawordlist
There’s nothing wrong when doing this, but what if we want to make sure that there are no duplicate entries in, for example, megawordlist_wpa2 from above. Here’s a quick command that we can run to make sure.
sort megawordlist_wpa2 | uniq -d
Depending on the size of the file, this command will take a long time to execute, but it will print any and all lines that are duplicates. There may be none (in which case you can skip to the next section), or there may be tens, or hundreds, or thousands of duplicate lines.
If you have duplicates, there’s a quick and easy command that can get rid of them all.
awk '!seen[$0]++' megawordlist_wpa2 > megawordlist_wpa2_nd
In this example, megawordslist_wpa2_nd would contain all of the same entries as megawordlist_wpa2, but with all of the duplicates consolidated to a single line in each instance, resulting in zero duplicate lines.
The only limitation here is RAM. awk needs to load the entire megawordlist_wpa2 file in to memory in order to sort and determine which lines are duplicates. If your text file is 16GB in size and you only have 2GB of free RAM, the command will fail. Keep this in mind.
Conclusion
I hope the examples in this article were helpful to you, and will save you some time in the future. If you have any commands that you’d like to add, put them in the comments below.
If you enjoyed this tutorial and would like to see more, please feel free to share this article on social media, comment below letting me know what else you’d like to see, and follow me on Twitter @JROlmstead.