PAN I Has Your hashes pl0x?

I thought to myself, if I knock on a specific port (445) across the whole IPv4 space, will anyone knock back, possibly reactive, and if so, who, what, when where and why? possibly the same questions they had for me knocking on their port.

The Skinny

Set up an SMB Listener and scan the internet, you will get stuff... from Palo Alto Networks PAN Agent via GlobalProtect Portal's public facing IP (if configured wrong) - you can skip the rest if you dont need the story


This is kinda derpy, but the impact is high, so... you know, facepalm or just twitch and 'i shouldn't be surprised' carry on

The Setup

I need to be able to receive incoming connections and log them, because I'm knocking on 445 I'm going to use something that also listens on 445, there are a few tools we can use but these two are my go too:

I also need something to hit those ports will, nMap I love, but MassScan is perfect for speed and case purpose. (thanks Rob)

aaand I want to throw them in screen sessions so I can forget about it while I eat a delicious selection of hacking cheese and such so .. yeah, screen command

before I kick off Responder I disable any unnecessary services I don't need by editing the Responder.conf file to only accepting SMB, well I leave 'http basic/http ntlm but that's for something else.

[email protected]:/usr/share/responder# responder -I eth0

Alright, that's listening on my Linux machine that is now on the DMZ of my network let's get scanning: we want as much of the IPv4 space as we can and we are only interested in port 445 and we want to write it out to a json file and we want to exclude 255.255.255.255 (sorry supernets)

masscan 0.0.0.0/0 -p445 -oJ massscan445.json --exclude 255.255.255.255

Incoming

My hunch paid off, I'm receiving a lot of SMB-NTLMv1,SMB-NTLMv2 challenges.

Once IPv4 was complete I had a total of 422collected hashes ntlm1 and ntlmv2

Number of Hits
[email protected]:~/Responder/logs# ls | wc -l 424

Domain Admins -_-
[email protected]:~/Responder/logs# cat * | grep Administrator | wc -l 197

Dodgy bash... list IP Addresses from the log filenames
ls *.txt |cut -f3 -d - | sort | cut -f 1,2,3,4 -d . > AllIPS.Palo.pew.txt

Bulk Whois
[email protected]:/usr/share/responder/logs# netcat whois.cymru.com 43 < ~/Desktop/AllIPS.Palo.pew.txt | sort -n > ~/Desktop/AllIPS.Palo.Whois.done.csv So that's cool, I have details about the IP but what's going to validate this is perhaps an SSL certificate, if port 443 is open...

nmap -p443 --script ssl-cert -iL AllIPS.Palo.pew.txt -v

some sites provided good information in the SSL certificate but it's slipping into OSINT now and I think I'll leave that for now

Back to the collection data

there are some interesting user-names but there seems to be a theme...
out of 424 User-names 228 where unique, some of the usernames where Administrator (So, I guess I could have have domain admin, or someone else already does) some where users, some where clearly shared use service accounts like SCCM, but the majority where related to Palo-Alto-Networks Pan-Agent, this looks like ill configured firewalls.

What I think is happening

I think the firewalls (Palo-Alto Networks)PAN Agent is forgetting what side of the gateway it should be snooping around on, If you use a Palo Firewall with AD Integration, what is supposed to happen is this, a user joins the network (not the domain**) and once it's joined the network the Palo agent will log into the new machine with A (ideally) low privilege AD service account to pull information about the user and report back to the firewall to set up the permissions of that user based on their group entitlements, so if they are in a relaxed group they will have full internet, if they are in a risky group ... restricted internet

** Really should only authenticate against domain members, not network members as anyone can be on the network (and 'do stuff with the agent's hashes, but on the domain, there is a layer of auth and possibly more security controls (logging alerting what not)... anyway

I think that if the Palo is being used for internal and external firewalling, then when you hit port 445 (over the internet) the Palo Network Agent reacts to it and tries to log into the machine that issued the port ping/scan

The Problem with that is that it's possible to get a foothold in the AD once you have these hashes, they may be crackable or passable but mostly it's not great, we have domain information too, so huge information leak.

From Palo

If the firewall is configured as such, it will attempt to probe an endpoint whose User-Id is unknown at the time that it tries to send traffic through it. This is related to the comment above, in such situations, the firewall will probe an endpoint irrespective of whether the "knock" from the client is on TCP/445, TCP/5008, TCP/389, or any other port. The reason we don't see any probes in response to a TCP-SYN on TCP/443 is that WAN-facing GlobalProtect services are hosted on this port - those services respond to the TCP-SYN in a normal manner.

Typically, the firewall integrates with an Active Directory Domain Controller to read Security Event Logs from the Domain Controller. This operation, as you noted, does not require us to join the Active Directory domain. However, this model of integration does not enable the probing feature of User-ID, all it does it is communicate with the configured domain controllers and read their security event log.

In addition to this model, which, the vast majority of our customers use, User-ID optionally allows an administrator to enable WMI probing. This option, when in use, instructs the firewall to probe endpoints in specific network zones when they send traffic through the firewall. Therefore, this is a two-step manual process: An administrator needs to manually enable WMI probing
An administrator needs to configure User-ID to probe a WAN-facing network zone
See https://www.paloaltonetworks.com/documentation/71/pan-os/pan-os/user-id/configure-user-mapping-using-the-pan-os-integrated-user-id-agent#33331 , step #4 that outlines this process.


Email Palo Alto Networks

Let's see if it's a product problem or a configuration problem.

PaloAlto where very interested, I worked with them to identify where the issue resides and after 23 emails we came to the conclusion that it is a deployment wetware issue (miss-configuration) although they did push an update out a week later making it difficult to make the configuration mistake that 400+ organisations have fallen victim to by ... not RTFMing ? they also pointed me to a document by rapid 7 that looks like they did pretty much exactly what I did in the late end of 2014 here

There are questions around deployments, PoC to Live, and maybe worth reminding their customers about pitfalls or even providing a service that checks for this for them, if it's 5-10 people making the mistake they probably went out of their way to make it, but if it's 2-4* hundred, you might want to make it a bit more important of a message to convey. looking at the timeframes of our conversations they released a fix/update for the user-ID WMI Client Probing shortly after I'd spoken to them, possibly planned, maybe brought forward, but yeah... upgrade to PAN-OS 7.1 ;)

From Palo:

To address this situation, a recent software update carried product enhancements that automatically disable probes bound for WAN IP addresses. For those of our customers who are in possession of large swathes of Class A addresses, the product continues to allow them to leverage this capability, but in order to do so, an administrator is now required to configure the range of public IP addresses they expect the firewall to probe. See "User-ID WMI Client Probing" here https://www.paloaltonetworks.com/documentation/71/pan-os/newfeaturesguide/upgrade-to-pan-os-7-1/upgrade-downgrade-considerations#16385 for more information on this product enhancement.

Further, we recommend that a least-privilege account be used for the purposes of scanning a domain controller for event logs, or for probing clients based on WMI. Configuration articles such as https://live.paloaltonetworks.com/t5/Configuration-Articles/How-to-Configure-Agentless-User-ID/ta-p/62122 and https://www.paloaltonetworks.com/documentation/71/pan-os/pan-os/user-id/configure-user-mapping-using-the-pan-os-integrated-user-id-agent#33331 are available for our customers to use. We also recommend that options such as interactive logon be turned off on such accounts, so that they cannot be used by a malevolent actor to logon to workstations within a network.

Kinda Gutted that H.D Got their before me but also kinda happy I'm thinking similar (in this case at least), and that if there are hundreds of organisations in this position there is still effort to be done to reach them and have them understand the position they are in.

All in all it was exciting, and gave me pause for thought around defensive appliances,and the scale of miss configurations (that they exist and are common) if you have ever had any experience in appliances that use active directory credentials even on a local network, you will see that they very rarely do it security - most are not aware of the difference between domain and network membership and will try and authenticate with anything that has an IP. and that might be because during the PoC it's time precious and "security can come later" but to be honest, i've never seen it done properly - the documents referenced above are great, not just for PaloAlto's.

A note for the Red's

Pentesters/Red Teams: Looking to exploit this assuming you have collected the hashes via tools mentioned above, you'll need to crack the hash if you can't pass it, so ... John or Hashcat, assuming the account is a following a least privilege model,if the AD Admin is on point the account will be low privilege, but there is still this module in Metasploit however if the account is provisioned badly/shared with elevated rights then ... happy days you can probably do what you want!

Worth talking about because it's still not been fixed, and what I have seen is pretty dangerous.

Final thoughts

It would be nice to make sure windows credentials never traverse the internet that might be something they could fix, but for now it's more the administrator of the Firewall, so if you are moving to PaloAlto Networks from another product, just make sure your Firewall guy and your pen testing team are legit, and have support they need to maintain the right kind of thinking and time to understand the technology, technology is quick.

I provided Palo Alto Networks with all the raw data, and provided them with OSINT informed direction who might own the vulnerable devices, I tried to contact a few that where using domain admin credentials for their agent but it got tiresome quick.

Palo have been reaching out to customers, with the information I provided them, so maybe they have more time and resource to get that done. in the meantime, if you reading this thinking oh crap I have one of those, you now know how to check ... at least externally,... upgrade PAN OS.


Contacted PAN on the 27th June 2016, New Pan OS 7.1.3 released 6th July 2016