A Story About Password Auditing

Hi everyone, sorry I haven't blogged in awhile. Life gets busy and with the recent boom of generative AI I wasn't sure about sharing original content like this that will inevitably be scraped and used to train models. Anyway, on to the story.

I enjoy learning from platforms like TryHackMe and HackTheBox. One technique I learned was how to obtain Windows password hashes and crack them using rainbow tables and Hashcat permutation rules. I thought this was really cool and it got me curious, how good or bad were the passwords in production? I got permission from key stakeholders, grabbed the password hashes, and ran Hashcat to crack them. Immediately my screen was flooded with hundreds of very weak very easily guessable passwords. I was shocked, was this real? I tested a couple and confirmed they were indeed real passwords.

 

 

How was this possible? We had a password policy with complexity and minimum length requirements. I began digging deeper and discovered that it was easy to meet the complexity requirements with a weak password. If you have four minutes to spare I recommend this very relevant bit by comedian Michael McIntyre

 

 

Let's take a closer look at a fairly common password policy that:

1. Requires an uppercase letter
2. Requires a number
3. Requires a special character
4. Requires a minimum length of eight characters 
5. Must be changed every 90 days
   

 

Password123!

Welcome2020!

Ihatepasswords1!

 

These are all acceptable passwords according to the password policy. Even worse, with rule number five forcing passwords to be changed on a regular basis, I saw that people were lazy and will simply increment a digit so "Ihatepasswords1!" becomes "Ihatepasswords2!" or "Welcome2020!" becomes "Welcome2021!".

 

I felt a sense of regret having looked under this rock to discover such a huge problem, but I soon started working on a solution. I started by improving the password policy. If you read up on NIST Special Publication 800-63B it says some things like:

1. Elimination of complexity requirements: NIST advises against enforcing arbitrary complexity rules (e.g., requiring a mix of upper-case letters, symbols, and numbers). Instead, passwords should be checked against known compromised credentials and prohibited password lists.
2. Elimination of regular password resets: NIST recommends not forcing users to change their passwords on a regular basis unless there is evidence of compromise. Regular resets often lead to weaker password choices due to user frustration.
3. Preference for longer passwords: NIST encourages using longer passwords or passphrases, as they provide better security than shorter, complex passwords. Longer passwords are harder to crack and easier for users to remember.
   

So here's my plan:

1. Remove the complexity requirements
2. Increase the minimum length requirement and educate users about passphrases
   
3. Remove the requirement to force a password change on a regular interval, it should only be necessary when an external event requires it
4. Create a custom list of banned keywords that are industry, brand, and country specific
   
5. Install a password filter to check the password being set against a list of known compromised passwords (something like 20 billion and growing with every breach made public)
   
6. Setup a password auditing tool to run on a weekly basis. This is important because the list of "known breached passwords" is always growing and it gives us an opportunity to see if a password is in use that was recently added to a breach list. The filter will only check it at the time the password is set.
   

 

I am happy to say that after some time, the project was a success. It took a lot of time and effort. I had to coordinate with many disparate teams, learn new technologies, educate thousands of non-technical people, implement new standards, force a lot of password resets (and probably upset a few people), participate in dozens of meetings, write documentation, coordinate schedules, and work with vendors. What started as a curiosity took me down a road I was not prepared for, but am glad that I did because I learned a lot and felt like I made a significant contribution to improving security.

An Introduction To External Attack Surface Management

In the past year or two I have been learning a lot about External Attack Surface Management and I wanted to share some of my knowledge. As a short introduction, what is an Attack Surface? Imagine you lived in a house with lots of valuables and you were in a bad neighborhood. You would probably lock your doors and windows at all times, and have additional layers of security like a barbed wire fence and an alarm system. In this scenario, the attack surface would be any point of entry a burglar could use to gain access to your valuables. A clever burglar would find a weakness in the fence, and test all of the doors and windows to find the easiest way inside.

To verify that you are secure, you could test your defenses by trying to break in yourself, or hire a professional to do it for you on a regular basis. What if you had thousands of windows and doors opening and closing regularly, and you were in a neighborhood where all of your neighbors are burglars? This is closer to the reality faced in the internet world.

A house with many doors. Source

 

An insecure computer plugged directly into the internet with no firewall can be compromised in seconds. So how do you monitor your attack surface? The first step is to inventory everything that is internet facing. Find every IP Address, Root Domain, and Subdomain you own. If you have access to your DNS registrar you can obtain a source of truth list of every configured DNS record. If you don't, you can use the same techniques used by Bug Bounty hunters and use subdomain enumeration tools like Amass, subfinder, and massdns. This inventory of assets will likely change over time, so it would be a good idea to run automation that updates the inventory as new assets are deployed and old ones are decommissioned.

The second step is to do a use your inventory list and run a simple port scan with a tool like nmap, masscan, or naabu to see what ports and services are open to the public internet. For the best results this port scan should be done from a dedicated server. This is like testing to see if your doors and windows are unlocked and what's behind them. The port scan should return some results with fingerprint data that gives you an idea what may be behind each open port.

If it's an SSH, FTP, or Telnet service, try to connect with a client to see if it lets you try to login with a username and password. You can also often see the version of the service running and do an internet search for known vulnerabilities.

If it's a web server, you can try to load it in your browser, or use a tool like httpx to see what information is available. Try to connect using HTTP and HTTPS, and if you can connect using HTTPS take a look at the SSL Certificate to learn more about the service. There are some great command line tools for parsing SSL Certificate data such as SSL Checker and some more mentioned by Jason Haddix in his recent Tool Time video discussing SSL Certificate Parsers. Explore the web server in depth to gain an understanding of what it's used for. Review the HTML and Javascript source code. Does it have any login forms where a username and password could be used? Does it reveal any information that shouldn't be public? Does it have any known vulnerabilities that could be exploited? Can you do Content Discovery to find hidden content?

For other open ports you will need to do some research to better understand what service is listening and what risk that could present by remaining publicly exposed.

As a general rule of thumb, you want to reduce your attack surface to improve your security posture, or in other words, the less doors and windows you have to lock, the less likely you are to lose your valuables. When you find an open port and service, you should ask yourself if you really want that to remain public, or if it should be hidden from public view. If you want to hide it from public view you will need to identify where it is hosted, and change the firewall settings to only allow access from specific management IPs, or close the port entirely. If it's an old and unmaintained asset, consider decommissioning it to save money and reduce your attack surface.

If you discover new assets are regularly deployed in an insecure state, identify who is deploying those assets and get to know more about their needs and processes. They may not know that the new servers they deployed all had a web server with an admin login page exposed on port 8000. Work with them to improve their processes so they are less likely to expose a risk to the environment in the future.

The steps here can all be automated, and depending on your budget, you can build a solution yourself or you can pay a third party to do it for you. Here is a nice list of free and premium solutions.

This is a high level introduction to the world of External Attack Surface Management, or as it is also known, simply "recon". I hope someone has found this information useful, have a great weekend everyone!

Bug Bounty: Behind The Scenes

As a Bug Bounty hunter it can be frustrating to submit a report and have to wait for a long time to get a response, have your report closed as duplicate/out of scope, or receive a reply from the triage team requesting a better explanation of the impact and to improve your PoC (proof of concept). Early in my career I was a Bug Bounty Triager and I wanted to write this to provide some insights on what happens behind the scenes when a Bug Bounty report is submitted.

Before a Bug Bounty program is created, many things need to be in place. The security team has to:

• Be very familiar with what public facing assets the company has
• Be very familiar with the development and infrastructure teams
• Know who to contact depending on what issues are reported
• Have the skills to understand the severity and verify the report is valid
• Have the customer service skills to interface with the hacker in a friendly manner
• Have support from senior leadership

Without these fundamentals, a Bug Bounty program is unlikely to be successful. To put these in another perspective, when I received a new report, these are the questions I would ask myself:

• Is the reported issue in scope for the program?
  
• Can I validate the issue is accurate with the provided PoC?
• What is the impact to the business if this issue were exploited by a Black Hat Hacker?
  
• Who do I contact internally to get the issue fixed?
• How does that team like new tasks to be added to their backlog?
• How do I explain and demonstrate the risk so they understand it?
  
• How will they prioritize the issue with their other tasks?
• Are there compensating controls that can be used to mitigate the risk?
• Could this issue exist in other areas?
  

Assuming the report is valid and in scope, I would normally create a ticket for another team to make a change and fix the issue. Depending on the workload of that team and priority of their other tasks, it could take them weeks or months to complete the fix. During this time I would often receive new messages from the Bug Hunter asking for an update, and I would do my best to be friendly and provide realistic expectations. Every Bug Hunter was unique, and some were understanding, while others would get frustrated and threaten to make the issue public if they weren't paid in a timely manner. This is part of the reason why many Bug Bounty programs are invite-only and curated so only respectable Bug Hunters participate in the program.

It wasn't uncommon for service owners to contact the security team and ask about unusual activity. Sometimes the scanners that Bug Hunters use can negatively impact services. If we were able to identify which Bug Hunter is causing the negative impact we would contact them and ask them to rate limit their scans, or simply ban their IP to protect the service. When reviewing logs it is not always easy to tell the difference between a Bug Hunter and a malicious hacker. This is why sometimes programs request that a unique user-agent is used, so it is easy to attribute scan activity with a specific Bug Hunter.

When the report involves a Remote Code Execution vulnerability, or the PII of customers/employees, it is typically treated as a P1/Critical issue. This is when the security team kicks into high gear and performs Incident Response. To me, conducting Incident Response was like being a "Digital Firefighter". Adrenaline kicks in and you have to try to think clearly and contain the issue as quickly as possible. Leadership must be informed, on-call people sometimes need to be paged, logs need to be reviewed, a timeline must be created, and other priorities take a back seat until the issue is contained. In this context, containment means that the risk is mitigated and can no longer be exploited.

Incident Response: Digital Firefighting

In June of 2021 Lesley Carhart posted an excellent photo of her heart stress during an incident:

Source: https://twitter.com/hacks4pancakes/status/1407165004424507401

Once an issue is fixed, I would message the Bug Hunter to ask them to verify the issue is fixed. Once everyone agrees it is resolved we would reward the Bug Hunter. The reward amount was often a discussion point within the team and many factors were considered. Primarily: "What is the impact to the business if this issue were exploited by a Black Hat Hacker?".

The work I did as a Bug Bounty Triager early in my career taught me a lot about Application Security, Penetration Testing, Incident Response, and Vulnerability Management. It inspired me to dive deeper into these topics and eventually learn how to do some Bug Hunting of my own. In May of 2021 I received my first paid bounty for two findings I submitted through HackerOne.

I am happy to see that Bug Bounty programs are being created by so many companies out there. It is a great opportunity for companies to improve their security, and for ethical hackers to get paid for doing what they love.

If you are interested in becoming a Bug Hunter, or just like to learn how hackers do their thing, I highly recommend these learning resources:

HackerOne CTF https://ctf.hacker101.com/

PortSwigger Web Security Academy https://portswigger.net/web-security

HackTheBox https://www.hackthebox.com/

TryHackMe https://tryhackme.com/

PentesterLab https://www.pentesterlab.com/

Many of these learning sites have built-in guides or you can find high quality guidance on Youtube

Advice For My Younger Self

Looking back at the beginning of my career in InfoSec, I made a lot of dumb mistakes. I would ask questions that could easily be Googled, I would skip over the fundamentals and go right to the advanced stuff, and I rarely read the manual (aka "RTFM"). Sometimes I find myself doing these things now, but at least I recognize it and work to improve.

Here is some advice I would give to my younger self that I hope someone finds useful:

Computers, networks, and software is complex and no one knows everything. Don't treat your senior team members as if they have all of the answers. When you encounter a problem, spend some time trying to solve it on your own. When that fails, read the manual, and when that fails, Google it. You might be surprised to learn that senior professionals Google everything.

Remember the human. Everyone makes mistakes, everyone has bad days, and everyone forgets things. Don't treat people like robots that should be perfect. This also applies to yourself, leave room for failure and learning.

It's not you versus me, it's you and me versus the problem. It's easy to encounter a problem and present it to someone else to solve, but it is more important that you treat people as if they are on the same team and collaborate on a solution.

Some conversations should not be done through text (email or instant message). It is possible that someone reads your email in a different tone of voice than you intended. When possible, converse through a face to face medium and use email to share supporting information.

As you learn things, take notes as if you will forget everything tomorrow. I like to write documentation because it helps me confirm what I've learned and share it with colleagues, as well as my future self when I forget some of the details.

Don't let perfect be the enemy of good enough for now. It's easy to get stuck trying to perfect something, when sometimes all you need is something that gets the job done for now. It is easier to improve something that already exists.

Don't be overwhelmed by how much you have to learn, take it day by day and follow what interests you. This image is just a sample of the subjects within Cybersecurity, each one can be a specialty on it's own. It is important to keep learning and challenging yourself, but you don't have to know everything. Once you develop a specialty it is relatively easy to move between others. 

Security has many specialties

Learn and respect the fundamentals. On the technical side of Cybersecurity you should know some basics about networking, systems, and coding. You don't have to be an expert at all three, or even know how to code, but the more you learn about these the better off you will be in your career.

Remember "CIA": Confidentiality, Integrity, and Availability. This is another fundamental concept that is at the core of all Cybersecurity careers. Our job is to understand and protect all three. Sometimes that can be as easy as verifying the data is backed up, or that an encrypted protocol is being used for transport.

Try, fail, and repeat until successful. Failure is part of the learning process, and I remember giving up after failing once. Get back on that horse as many times as it takes!

Certifications can be helpful to give you some structured learning, but they alone will not make you successful. You can use certs to confirm your knowledge, improve your resume, or to help you pivot into a new specialty. Everyone learns different and sometimes it's better to learn on your own.

Immerse yourself in the hacker culture and connect with the community. Listen to Darknet Diaries, read /r/netsec, watch Hackers, follow security professionals on Twitter, Setup a Feedly account to monitor RSS feeds for blogs and news, read popular books (fiction and non-fiction), go to meetups, join a discord, try HackTheBox, and share your code or ideas to give back to the community.

Understand the difference between knowledge and experience. It takes time to obtain experience, you cannot rush it. Enjoy the journey!

I think that's enough for now, I'll leave you with some advice from Professor Feynman:

• Study hard.

• What others think of you is none of your business.

• It's OK not to have all the answers.

• Experiment, Fail, Learn and Repeat.

• Knowledge comes from experience.

• Imagination is important.

• Do what interests you the most.

• Stay curious.

Recent Accomplishments

Not long after my last blog post I was hired as a professional penetration tester for the first time in my IT Security career. This had been a dream of mine for some time, so I was full of excitement and motivation to learn everything I could. I completed the eLearnSecurity Junior Penetration Tester (EJPT) certification in May and immediately started working on my next cert, the Certified Red Team Professional (CRTP) from PentesterAcademy. I quickly learned that although the description says it is for beginners, it actually felt quite advanced for me in the beginning. I even took a Udemy course called Advanced Scripting & Tool Making using Windows PowerShell to improve my comfort level with PowerShell. The CRTP course focuses on attacking and defending Windows Active Directory. Some of the things you will learn as a student:

• Active Directory Architecture
• Kerberos
  
• Active Directory Enumeration
• Local Privilege Escalation
• Domain Privilege Escalation
• Domain Persistence and Dominance
• Cross Trust Attacks
• Forest Persistence and Dominance
• Defensive Monitoring
• Bypassing Defenses
• Deception Techniques
• PowerShell
• Mimikatz
• BloodHound
• PowerView
• Custom Obfuscation
  

Just yesterday I submitted my final exam report, and hours later I received confirmation that I passed! For me this is the result of four months of effort and hundreds of hours of practice. In addition to the CRTP I also completed these Windows Active Directory challenges:

Practice Pentest Report - HackTheBox - Postman

I have been studying for my OSCP certification recently and purchased VIP access to HackTheBox.eu. This site is an excellent resource for penetration testers of all levels. For those that aren't aware, HackTheBox is a penetration testing lab with live machines to practice your hacking skills against. In the beginning you are only given an IP address and have to figure out how to gain access to the "flags" which you can then submit to the leaderboard for points. So far I have only completed a couple of the "easy" rated boxes, such as one called "Postman". After completing Postman I took the opportunity to practice writing my first "penetration testing report" for a fake company. Normally after a penetration tester is hired, they deliver a report to the business that hired them to review the discovered issues and recommended follow-up actions. Here is my Postman "Pentest Report":

Introduction

A “penetration test” was requested by “HackTheBox” for their soon-to-be-deployed “Postman” service. The goal of this test is to verify that security is up to par with their expectations before being released to production. The test will be done in black box format, without access to any code or prior knowledge of the system.

Scope and Duration

The scope of this test is limited to the “Postman” server located at 10.10.10.160. The duration for the test is one week starting from the 18th of November 2019. The client has requested we avoid using any "DoS" exploits to avoid unnecessary crashes. If root level access is obtained, the testing can be stopped immediately and the test results reported to the client for remediation.

Technical Summary

Initial port scanning revealed several exposed services, such as HTTP, SSH, Redis, and Webmin.

Initial Port Scan Results
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Discovered open port 6379/tcp on 10.10.10.160
Discovered open port 80/tcp on 10.10.10.160
Discovered open port 10000/udp on 10.10.10.160
Discovered open port 10000/tcp on 10.10.10.160
Discovered open port 22/tcp on 10.10.10.160
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Port Scan Results With Banners
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
22/tcp    open  ssh     OpenSSH 7.6p1 Ubuntu 4ubuntu0.3 (Ubuntu Linux; protocol 2.0)
|_banner: SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
80/tcp    open  http    Apache httpd 2.4.29 ((Ubuntu))
|_http-server-header: Apache/2.4.29 (Ubuntu)
6379/tcp  open  redis   Redis key-value store 4.0.9
10000/tcp open  http    MiniServ 1.910 (Webmin httpd)
Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Particularly interesting is the Redis service, which is an in-memory database application. Internet research reveals a known security flaw detailed by the developer here: http://antirez.com/news/96. While attempting to reproduce this we discovered that command line access to redis was easily obtained and commands can be executed.

We continue to follow the developers instructions and create a text file containing our public SSH key with newline padding. Once that is created we need to:
1. Change the dbfilename to authorized_keys
2. Change the config directory to .ssh
3. Insert the contents of our text file into a “set” within the database
   1) "Redis Sets are an unordered collection of unique strings. Unique means sets does not allow repetition of data in a key."
5. Save
6. Test to see if we can SSH in

We can see that this attack was successful, and we now have a shell on the system as the redis user. This user only has limited access:

While searching the system for files containing sensitive information, we discovered a file called “id_rsa.bak” in the /opt/ directory. The owner is shown as the “Matt” user and the contents reveal it is a private key.

Internet research reveals an article ( https://medium.com/@canarerobertjohn/basic-pentesting-2-325cafc19fcc ) with instructions for cracking a private key such as this to reveal the passphrase. The steps are as follows:

1. Convert the key to a format supported by JohnTheRipper
   1) "John the Ripper is a fast password cracker, currently available for many flavors of Unix, macOS, Windows, DOS, BeOS, and OpenVMS"
3. Run JohnTheRipper on the converted key using a wordlist

Sorry for the small text...

Within seconds we have revealed the passphrase, and using this information we want to see if it's possible to escalate our privileges to the Matt user.

The privilege escalation was successful, we are now logged in as Matt and have more access, but not to everything.

Additional testing reveals that the credentials for “Matt” also work for the webmin panel:

Earlier banner information revealed Webmin as being version 1.910, and searching for exploits reveals one:

Now we load up MetaSploit with this exploit and configure all of the options:

After running the exploit we have successfully obtained root level access to the system.

Remediation Recommendation

This server was able to be fully compromised due to a series of weaknesses that were easily exploited. We recommend:

1. Closing ports to services that do not need to be publicly exposed
2. Removing banner information that can reveal software versions
3. Training your users not to leave sensitive information on the server in a directory readable to all users
4. Enforcing stronger password requirements
5. Patching all software to the latest version

Improving Security With Kindness

A couple years ago I was on a flight next to an older woman who had never flown before. She had no idea how to use the tablet in front of her, or even how to buckle her seat belt. She asked me for help and I think at this point some people would sigh and dread sitting next to her for the rest of the flight. However I saw this as an opportunity to help someone who was technically illiterate, and probably nervous about her first flight. After I helped her buckle up I showed her how to plug her headphones into the tablet and start a movie, adjust the volume, play a game, or some music if she wanted. She was a sweet grandma and told me all about her grandkids and her life in small town America. Throughout the flight she would ask me questions and I took my time to explain everything as simply as possible. After the flight while waiting in line one of the passengers who sat behind me tapped me on the shoulder and asked how I could deal with someone like that, and said I have incredible patience. I said it was no problem and that I was just happy to help.

The way I see it, I am fortunate to have grown up around computers and be very comfortable with technology. However, not everyone is as fortunate or naturally inclined to learn how to use tech. I think it's even safe to say there are a lot of people out there that are confused or even afraid of tech. These are the people who still cling to old software, POST IN ALL CAPS, and always seem to get viruses. This is where I say that it is our job, as the technically literate, to exercise our patience and kindness to help someone who is struggling with something we figured out years ago.

Early in my IT career I worked at a Helpdesk where we were each assigned trouble tickets to work on. These tickets were often things like installing a printer, migrating someone's email to a new device, or replacing a broken hard drive. My colleagues would often cherry pick the tickets from users who were easy to work with, and avoid the "problem users". I had no problem taking on the "problem users" tickets, seeing them as opportunities to help someone in need. Many of these users were experts in their fields, and very well educated, but when it came to technology they were frustrated, confused, and angry. I often thought to myself that those are normal emotions felt by all of us when learning something new, or when something "just doesn't work", and that these same people would gladly help me if I were frustrated with something they were experts at, so why don't I share the same courtesy?

Free Cookies!
(Image Credit: Wikimedia)

I would approach each of these users with some friendly small talk, try to get the backstory on what's going wrong, empathize with their frustration, and start working on solving the problem while doing my best to explain what's going on. I would often recommend they take notes to reference later when they run into the same problem again, and come up with small tricks like using a Eudora skin on Thunderbird to make them more comfortable with a new interface. Over time these users encountered fewer problems, submitted less tickets, and some even developed a new appreciation for technology and started learning more on their own. At the end of the day this meant less work for my team, much happier users, and sometimes free cookies for me :).

More recently in my career I was building a Python script that had to be used by some of my technical and non-technical colleagues. While testing my first version of the script with my technical colleagues they told me how great it was and thanked me for building it. However when testing the same script with my non-technical colleagues, they expressed concerns about the difficulty of use and risk of making a big mistake. In my original design I had not considered this perspective, but it made sense when brought to my attention. I improved the script to make it more simple to use, added confirmation prompts where necessary, and expanded the documentation. I went back to my non-technical colleague and had them test it, they were thrilled at how easy it was to use, how hard it was to make a mistake with the confirmation prompts, and appreciated having documentation to reference.

Anyone working in IT Security can tell you that the weakest link is always the human. Phishing emails work because they trick people into installing malware. Hackers breach databases because people make mistakes in configuring the systems that are supposed to protect it. People use old vulnerable software because it's what they are used to. When these things happen we need to:

1. Avoid shaming, that doesn't help the situation
2. Be sympathetic, everyone makes mistakes
3. Be patient, not everyone is on your level of comfort, knowledge, and experience
4. Work with them to understand the problem from their perspective
5. Educate without condescending
6. Support

I honestly believe that if these steps are taken, hackers will have a harder time doing bad things, and the internet will be a safer place for everyone.