Archive
Code: PDF hunter
So of late I’ve been playing around a lot with Scapy and pcap files, mostly for my sniffMyPackets project but also because it teaches me more about network forensics and python. The other area I’m starting to learn about is Malware Analysis and I’ve been spending some time looking at the Honeynet Project challenges.
One of the challenges to is to find the malicious content within a PDF file that is provided to you in a pcap file. Normally I would just reach for Network Miner and rebuild the file(s) that way but I wanted to see if I could write some code myself.
The goal of my code was simple, parse through a pcap file, identify a PDF and then rebuild the file so that if a tool such as exiftool or file was used that it would correctly be identified as a PDF and that you could open the PDF and view the content (if you wanted to).
I follow a certain process when I’m carving up pcap files, it’s not rocket science really just common sense. First off find the packets you are interested in, I tend to use a mix of Wireshark and Scapy for this and then look for something you can use to filter down to the packets you want before getting into the nitty gritty of carving them up.
For this piece of code I need to find some way of identifying a PDF file in a pcap file and as most PDF files will appear in a pcap file as part of an HTTP conversation, I parse each packet and if the packet has a Raw layer (a raw layer in Scapy is essentially the payload of a packet) then I look for this ‘Content-Type: application/pdf’. If this is matched then I store the TCP ACK number as a variable for use later.
Now once I have the ACK number I then need to find all the packets that relate to this in order to get the whole file. Now it turns out the ACK is the same for all the packets that the PDF download is in (something I didn’t realise until I started this) so it’s a simple case of using the following code to find all the packets I’m after:
for p in pkts:
if p.haslayer(TCP) and p.haslayer(Raw) and (p.getlayer(TCP).ack == int(ack) or p.getlayer(TCP).seq == int(ack)):
raw = p.getlayer(Raw).load
cfile.append(raw)
If either the TCP ACK or SEQ match our stored ACK variable we get the Raw layer and store it into a python list. This means that we now have (hopefully) all the packets that make up the PDF stored nicely away and because it’s a TCP conversation they should all be in the right order.
Now that we have all the packets we write those out to a temporary file, it’s a temporary file because if you were to open it in a text editor you would see all the HTTP headers at the top and the bottom, which means if you ran file against it, then you would get back a file type of “data” and not “PDF” (which is what we are after).
So we then have to do some python magic (well I think it’s magic), to slice the rubbish out. Now this is the part that took me the longest to figure out. If you have ever looked at a PDF file in a text editor (I wouldn’t blame you if you haven’t), you would notice that they start with “%PDF-” and end with “%%EOF” so finding the start of a PDF file is easy, the problem is that a PDF file can have multiple %%EOF towards the end of the file and I kept cutting at the wrong point.
To fix this I came up with a bit of a long-winded way of carving the temporary file up (see the code below):
# Open the temp file, cut the HTTP headers out and then save it again as a PDF
total_lines = ''
firstcut = ''
secondcut = ''
final_cut = ''
f = open(tmpfile, 'r').readlines()
total_lines = len(f)
for x, line in enumerate(f):
if start in line:
firstcut = int(x)
for y, line in enumerate(f):
if end in line:
secondcut = int(y) + 1
f = f[firstcut:]
if int(total_lines) - int(secondcut) != 0:
final_cut = int(total_lines) - int(secondcut)
f = f[:-final_cut]
outfile2.writelines(f)
outfile2.close()
else:
outfile2.writelines(f)
outfile2.close()
If you read Python awesome, if you don’t here’s what happens.
First off I open the temporary file and count the number of lines, I look for the variable I declared at the start of the code as start (which is this: start = str(‘%PDF-’)), if that’s matched it stores the line number as the variable firstcut
I then need to find the last cut, I look for the variable end (which is this: end = str(‘%%EOF’)) now remember I said a PDF can have multiple EOF statements, well I get round that because Python overrides the variable secondcut each time it’s matched so the last line with EOF is always the one used. I also add a +1 to the line number because for the next chunk of code if I didn’t I would actually cut the final %%EOF file the file (I know this because I did it, before realising what was happening).
So we now do a simple little IF statement to make sure that there is something at the end of the file to cut (sometimes there isn’t on the pcap files I’ve used/made) and if there is we slice the bad HTTP headers out before saving the file. If there isn’t anything to cut then we just save the file.
Hopefully that makes sense to non-python people (I can but hope).
I’ve tested this on a number of different pcap files that have PDF downloads in them and it works, I can open and view the PDF and if I run file or exitfool against it then it appears as a normal PDF. I’m sure there are some cases when it won’t work 100% but if you find something that doesn’t let me know so I can try to fix it.
The code can be found here: https://github.com/catalyst256/PDFHunter (in my ever-growing GitHub repo). Oh and I’ve added this function into my sniffMyPackets transform pack.
Enjoy!
Popping my Netwars cherry
So you may remember from earlier blog posts (before all the OSCP stuff) that I had started taking part in the Cyber Security Challenge UK contests. These are security related challenges that are aimed at providing cyber learning opportunities and possibly career opportunities to people with a passion for all things InfoSec related that aren’t currently working in the field.
I’ve signed up for a few of these but only submitted a couple due to the others falling right in the middle of OSCP time (which was more fun..). Now my aim for taking part was never to win, its just that I like doing these sorts of challenges and in the UK there is a distinct lack of CTF type things for a wannabe like me to take part in (more on that in later posts).
For both the challenges I submitted I scored OK, not 1st but not last either and like I said it was fun and I got to learn a few new things. Needless to say I was a bit surprised when I got an email inviting me to a special SANS Netwars event organised by Cyber Security Challenge UK, but as it had the word “Netwars” in it who am I to turn it down..
I arrived at the event this morning, with 29 other contestants, all various ages and background all of us with one thing in common, we all have a passion for InfoSec. The 8 highest scorers of the day would get a place on a “Masterclass” event in March 2013.
For those that don’t know what SANS Netwars is all about I suggest Google (we’ve talked about this before). Netwars events are “open book” which mean you can take whatever tools you want (Backtrack, Backbox etc. etc.) and the aim is simple, score points.
The actual Netwars was scheduled to run for 2 hours and you have to progress through 5 levels, the first 2 levels are achieved by using a bootable ISO image provided by SANS, level 1 is done as a normal user, level 2 is done with root priveleges. Both level 1 and 2 are more about forensics, things like “look at this pcap file and sha1 hash the IP address that made a DNS request to…” or “crack the password for root by using this backup file of the /etc folder”.
Now I like forensics but I suck at it, although I managed to get enough points for each level I did struggle with somethings, but I was surprised with some of the answers I managed to work out and a lot of that I think was due to doing the OSCP course. Level 3 is where (for me) the fun should start, attacking machines in a DMZ environment and finding the necessary “flags”, unfortunately by the time I got to level 3 I had about 20 minutes left so didn’t manage to score any points.
Just for information, level 4 is pivoting from the DMZ network to the “intranet” (again something I would have enjoyed and am well practiced at) and level 5 was “King of the castle” where you get to defend against other hackers.
The good and the bad:
Good
1. It was fun, by now you should guess that I love this kind of thing and would rather be doing offensive than defensive security.
2. SANS organised and ran the event incredibly well, the instructor/guide James Lyne was funny, helpful and helped make the event great, although playing Gangham style was a bit of a distraction (and it’s stuck in my head still).
3. See point 1
Bad
1. It wasn’t long enough, yes I know it was free and I should be grateful but, like I said I love this kind of thing so it would have been nice to see how far I could have got with a few more hours.
On top of this I got to meet a few people, spread the word about B-Sides London, drink nice coffee and I think I managed to avoid the TV cameras for most of the day, although there are some photo’s of me somewhere…
Now SANS very nicely have said they will email everyone their scorecards from the event which once I have I will post, but in the mean time here’s a screenshot at the scoreboard at the end of the day…
So I’m through to the Masterclass in March, I have no idea what that means or involves but once I do I will let you know. Between now and then I have some areas I need to focus on to improve my skills but as we all know its a never ending process really.
Penetration Testers are Cool??
A few days ago on twitter @markofu posted a tweet which at the time I didn’t give much thought to, but on reflection was actually true (especially in my case). The tweet went something like this.
“Hi..How do I get into security, I want to be a pen tester??” -> why?? “I dunno, cos it’s cool”
This post is all about my perspective on this subject, it is in no means set in stone and as always I’m happy to talk about it further if people wish (that’s what comments are for).
Now as this is my blog, this post will focus on me, well to be more accurate, on my interpretation of this tweet. So before we begin lets rewind the hands of time and take a journey into my past.
Many many many years ago I watched a movie called Sneakers, if you haven’t seen it then watch it if you have the time. It’s basically a movie about a team of misfits who get paid to break into companies and steal data (on behalf of said company), sound familiar?? Yep it’s a Red Team basically and back then it’s what drew me into IT and IT Security.
My first experience of InfoSec was when I was at college and I got my first computer (I was a late starter), I remember spending hours writing a Microsoft Word macro that would encrypt and decrypt text within a word document. In my first IT role (back in the days of hubs) I wrote a proof of concept paper on how a “hacker” could use a packet sniffer and collect the clear text telnet data that was sent to the Unix server, to take it even further I then described how you could “steal” money from that system.
Fast forward to 2012, and I finally get around to pursuing the world of InfoSec more (feel free to add any number of lame and half arsed excuses into why it took so long). I set myself a goal to not only learn more about Information Security but to also try to “break” into the tight-knit infosec community that seems (in my experience and opinion) quite daunting and “alien”. Over the last 7 months I think I’m made good progress, I’ve started this blog, wrote my Scapy guide, attending (well actually worked) at BSides London, taken part in UK Cyber Security Challenges and tried to learn as much as possible.
So what you may be asking is this all leading to and what does it have to do with the Tweet at the start??
You see just before Christmas @markofu was interviewed on @securityninja’s blog in a post called “Random Thoughts on Education & Learning“, I posted a comment asking for advice about how to “break into security” (that’s a well used phrase) and one of the suggestions was the InfoSec Mentor Project.
During the sign-up process (to be a mentee, and I’m still looking for a mentor), one of the questions is about what you would like be able to do once you get a mentor (well something along those lines) and I wrote “Be able to perform Penetration Tests”. Why?? Honestly because at the time I thought it was COOL. No I’m not joking either, the reason being is that for me being a Penetration Tester is the closest you can legally get to being a real life “hacker” (without getting into trouble).
I mean lets face it, what doesn’t sound cool about a job where you get to “hack” into other people’s computer systems and pit your skills and abilities across a never-changing landscape of firewalls, servers and network infrastructure??
The reality though is a lot different, the level of knowledge, experience and skill means becoming a pen tester isn’t as simple as taking a course or practicing in a test lab, it’s about real world, never-ending experience and that’s it’s not easy to just jump into it as a career and if you have an already established career it’s not always something you can achieve, without some sacrifices (there are always exceptions of course).
So here’s the twist (well sort of), I’m going to start my OSCP course soon, which will teach me essentially how to perform penetration tests (that’s still cool right?), however I don’t necessarily want to focus on pen testing, but courses like the OSCP are still very important for people like me because regardless of what area I chose to focus on, I still need to understand how attacks work and be able to test and verify things on my own (good solid foundation).
You see over the last few months I’ve discovered (on my journey to become a pen tester) that I actually have more interest in other areas of infosec that might not have come to light, if I wasn’t focused on learning about pen testing.
Make sense so far??
So once my OSCP course is done (and hopefully I pass) I’m going to change my focus from pen testing to one of the areas below (or maybe all of them):
1. Exploit writing – This still confuses the hell out me but being able to write code to exploit software seems like something I would enjoy getting more into.
2. IDS/IPS – Despite not being a packet ninja (or networking person by profession), there is something intriguing about being able to dissect network packets, identify attacks and understand how to stop them.
3. Malware analysis – To be honest I’ve just added this to my list (I wanted 3 items on my list), but again it’s one of those areas that I get to take apart other people’s work (the malware writers), analyse it and then work out how to identify and stop it.
I will of course still continue to build on the skills I’ve started to develop at the moment but I want that focus on one area (or 3) because for me that’s a better fit and I really do want to give something to the infosec community (not in a dirty way either).
So are Penetration Testers cool? Hell yeah, I’ve met a few, worked with a few and to be honest what they do on a daily basis is both scary and awe-inspiring at the same time, so I can understand why people associate pen testers with “COOL”, but sometimes you have to dig a little deeper to find your “niche”.
Adam
OSCP – My journey begins
You may remember that in some of my previous blog posts I mentioned that I was studying for my Security+ exam, well the good news is that I passed the exam nearly 2 weeks ago (there isn’t any bad news). The Friday just gone, I had a meeting with my big boss at work and he agreed that I could get funding for my next certification, what’s even better is that he was happy for it to be the OSCP (Offensive Security Certified Professional) course… 
It’s going to take a few weeks before all the necessary work related paperwork is sorted and I can start the course. Not wanting to wait I’ve already started researching and studying some of the areas I know I need to understand better before I start the course and I’ve started preparing the necessary “environments” so I can practice.
1. Download the PWB (Pentesting with Backtrack) syllabus, so I can work out my “weak” areas.
2. Build a virtual test lab so I can practice, this includes an installation of Backtrack 5 R2, Windows XP, Metasploitable 2 and LAMP Security VM’s. I am using VMware Workstation (perk of being a VCP) so I can configure multiple networks and all my VM’s run off a 32GB SD card in my laptop.
3. Read other people’s reviews of the OSCP exam just to get an idea of what to expect.
4. Practice my documentation skills in my lab environment.
Now there are a couple of things that both worry me and excite me about this course (and the exam), I’ve read some post OSCP reviews and the general theme is “It’s not easy”, but you learn a LOT, the other thing that worries me is the report writing that you MUST complete at the end of the exam in order to pass. Now I’ve never written a report like this before so for me it’s a worry, not something I can’t overcome, I just need to practice and get familiar with.
In a weird way I’m actually really looking forward to the exam, 24 hours to break into a number of hosts using all your skill, imagination and determination sounds like awesome fun. I’ve already accepted that I won’t sleep during those 24 hours but I don’t care (the joys of working overnight means it doesn’t bother me much anymore). I’ve also accepted that leading up to the exam I will spend many many many hours in front of my laptop studying and practicing but again to me that’s fun as well..
I obviously can’t post anything about the actual course or the exam content but I will post updates on how I am doing and if I find a new technique along the way I will try to post it up on my blog.
Just remember if first you don’t succeed, “Try Harder..”
Are you up for the challenge??
Hello, apologises for the lack of posting I’ve been revising for an exam (which is tomorrow), regardless of that yesterday I took part in my first Cyber Security Challenge UK, I’ve never tried this sort of challenge before so was a bit nervous as to what to expect, in the end I scored 24/39 which I’m happy with to be honest. I learnt some new stuff and enhanced some of what I had locked away in my brain and it was awesome fun.
The only problem is that I want to do more of them, not only are they fun but it’s a great way to learn. The problem is in the UK there don’t seem to be many “hacker” challenges for people to take part in. Then tonight I was tweeting with @tamonten (who just beat my score on the challenge in a shorter time period…)
I then had an idea (yeah dangerous I know), what’s to stop me from creating some challenges for people to try, don’t get me wrong I’m not a security “expert” (just felt a bit sick typing that) but then I remembered a saying one of my old martial arts instructors use to say “Don’t be shy, give it a try” (just before I ended up on the floor in pain).
So once the evil exam of doom is out of the way (it’s not that bad I just hate exams) I’m going to start trying to come up with some security challenges, just small ones to start with then maybe moving onto bigger things. I think it’s a good way for me to learn stuff and, well to be honest as long as I’m having fun that’s all I care about..
As always this isn’t just about me (honest), it’s about the community so if you have an idea (but not the time) for a challenge let me know (full credit will go to you) and I will try and make it happen. I am going to try and cover a few areas but like I said I’m still new at this so might take me a while to get started.
Adam
Scapy Guide: Bad checksum, naughty checksum
These posts will eventually make it into the guide but I don’t have the time currently so I will do a series of short posts as and when I can.
Today we are going to look at packet checksums and how to ensure that when you are using packets within Scapy that you don’t send packets with “bad checksums”.
Wikipedia has this to say about checksums:
The checksum field is the 16 bit one’s complement of the one’s complement sum of all 16-bit words in the header and text. If a segment contains an odd number of header and text octets to be checksummed, the last octet is padded on the right with zeros to form a 16-bit word for checksum purposes. The pad is not transmitted as part of the segment. While computing the checksum, the checksum field itself is replaced with zeros.(sourced from http://en.wikipedia.org/wiki/Transmission_Control_Protocol#Checksum_computation)
So how does this relate to packets we create or re-use in Scapy?? Well lets start at the beginning. First off we are going to create a simple packet for testing:
>>> pkt=(IP(dst="10.1.99.2")/ICMP()/"HelloWorld")
We’ve given this packet a name of pkt so we can reference it easier later on. So lets look at it’s default values.
>>> pkt.show()
###[ IP ]###
version= 4
ihl= None
tos= 0x0
len= None
id= 1
flags=
frag= 0
ttl= 64
proto= icmp
chksum= None
src= 10.1.99.25
dst= 10.1.99.2
\options\
###[ ICMP ]###
type= echo-request
code= 0
chksum= None
id= 0x0
seq= 0x0
###[ Raw ]###
load= 'HelloWorld'
Notice how the chksum value in both the IP layer and the ICMP layer show None? That’s because the show() function doesn’t show the checksum values of a packet before it’s sent (the checksums are generated by Scapy when the packet is sent)
If you want to see the actual checksum value for that packet when it gets sent, you need to use a different Scapy function. Today we are going to use show2().
Lets have a look and see what the difference is:
>>> pkt.show2()
###[ IP ]###
version= 4L
ihl= 5L
tos= 0x0
len= 38
id= 1
flags=
frag= 0L
ttl= 64
proto= icmp
chksum= 0xa0b9
src= 10.1.99.25
dst= 10.1.99.2
\options\
###[ ICMP ]###
type= echo-request
code= 0
chksum= 0xf7ff
id= 0x0
seq= 0x0
###[ Raw ]###
load= 'HelloWorld'
Now we can actually see the chksum value for each layer. Lets send the packet (pkt) to Wireshark.
>>> wireshark(pkt)
This should load Wireshark in a seperate window and display the ICMP packet, if we expand the IP and ICMP layers within Wireshark we can verify the checksums are correct against what we saw above in the show2() output.
Header checksum: 0xa0b9 [correct] (IP Layer)
Checksum: 0xf7ff [correct] (ICMP Layer)
OK so lets get to the bad checksum part of this post, we are going to save this packet to a pcap file and then reload it back into Scapy. We are doing it this way so I can show you a more realistic reason why you need to “sort” bad checksums.
>>> wrpcap("/tmp/ping.pcap",pkt)
>>> t=rdpcap("/tmp/ping.pcap")
>>> t.summary()
IP / ICMP 10.1.99.25 > 10.1.99.2 echo-request 0 / Raw
The first command writes the pkt to a pcap file in my /tmp directory, we then re-read it as t and display a summary to show it’s the same packet. Once we are happy with that we store the single packet as bad and show it’s values
>>> bad=t[0]
>>> bad.show()
###[ IP ]###
version= 4L
ihl= 5L
tos= 0x0
len= 38
id= 8
flags=
frag= 0L
ttl= 128
proto= icmp
chksum= 0x60b2
src= 10.1.99.25
dst= 10.1.99.2
\options\
###[ ICMP ]###
type= echo-request
code= 0
chksum= 0xf7ff
id= 0x0
seq= 0x0
###[ Raw ]###
load= 'HelloWorld'
Oh what’s that you say, you used show() and saw the chksum value?? Yes well spotted, bad checksum’s only seem to be an issue when you are reading packets from another source (i.e. a pcap file). If you generate new packets within Scapy the checksum is only generated when you send it, which means changing the values on our original packet (pkt) doesn’t generate a bad checksum message (before you send it).
Lets change the TTL again and then send the new packet (bad).
>>> bad.ttl=213
>>> bad
>>> wireshark(bad)
This is what I saw in Wireshark (and you as well I hope).
Header checksum: 0x60b2 [incorrect, should be 0x0bb2 (maybe caused by "IP checksum offload"?)
Yep seems that my checksum is all messed up, this means that the packet (bad) is not really any good for anything and you can't send it due to the bad checksum. So how do we deal with this??
First off, lets wipe out the old bad packet and recreate it again.
>>> bad=t[0]
Then we can quickly check the chksum value currently stored:
>>> bad.show()
###[ IP ]###
version= 4L
ihl= 5L
tos= 0x0
len= 38
id= 8
flags=
frag= 0L
ttl= 213
proto= icmp
chksum= 0x60b2
src= 10.1.99.25
dst= 10.1.99.2
\options\
###[ ICMP ]###
type= echo-request
code= 0
chksum= 0xf7ff
id= 0x0
seq= 0x0
###[ Raw ]###
load= 'HelloWorld'
Now before we change the TTL again, we need to delete the old chksum value so that it gets recreated again after we modify the TTL.
>>>del bad[IP].chksum
Now we change the TTL:
>>> bad[IP].ttl=222
Do a quick check to make sure that’s worked:
>>>bad.summary
Now if we look at the stored chksum value in the packet (bad) what do you notice?
>>bad.show()
###[ IP ]###
version= 4L
ihl= 5L
tos= 0x0
len= 38
id= 8
flags=
frag= 0L
ttl= 222
proto= icmp
chksum= None
src= 10.1.99.25
dst= 10.1.99.2
\options\
###[ ICMP ]###
type= echo-request
code= 0
chksum= 0xf7ff
id= 0x0
seq= 0x0
###[ Raw ]###
load= 'HelloWorld'
The old stored chksum value is missing. Now if we send this to Wireshark what happens?
Header checksum: 0x02b2 [correct]
Once again the checksum is good because it gets recreated when we send the packet. We can verify the checksums match by using the show2() function again.
>>> bad.show2()
###[ IP ]###
version= 4L
ihl= 5L
tos= 0x0
len= 38
id= 8
flags=
frag= 0L
ttl= 222
proto= icmp
chksum= 0x2b2
src= 10.1.99.25
dst= 10.1.99.2
\options\
###[ ICMP ]###
type= echo-request
code= 0
chksum= 0xf7ff
id= 0x0
seq= 0x0
###[ Raw ]###
load= 'HelloWorld'
Well I hope that all makes sense to you? If you are modifying existing packets that display a chksum value when you use the show() function, you need to delete them before sending the packets.
A general guideline would be if you modify a value in a layer that has another layer above it, delete the chksum value from both layers. Some layers checksum’s are generated from the information in the layers below them (if that makes sense). If in doubt just delete all the chksum values..
Adam
Scapy Guide – The Release
Two weeks ago I posted about my intention to write a “dummies” guide to Scapy. So here it is “The Very Unofficial Dummies Guide to Scapy”. If you have read the online version it’s still worth a look as I’ve made it look all nice and shiny and added some additional content that wasn’t in the online version.
The whole point of this guide was to write a beginner’s guide for scapy, and when I say beginners I include myself. Up until I starting writing this guide I hadn’t used Scapy so I’ve learnt as I’ve gone. The guide covers the basic Scapy functions and abilities, it won’t make you a Scapy expert but I hope it will give you a start.
I actually really enjoyed writing this, and using Scapy and as such I intend to carry on updating the guide. I’ve already thought of some more things to add into the next few chapters and once I’ve finished studying for my Security+ exam in June I will carry on working on this guide.
I’m not expecting the guide to be perfect, I’ve read through it a few times now, and made changes as suggested by the people who have proof read it for me. I only ask 2 things of the people who take the time to read this guide.
1. Please provide feedback and comments, good or bad I don’t mind and if you post them on this post I will make them public. It’s important for me to understand if you actually find this useful and if there is anything else you want to see in it.
2. I’m also still looking for ideas and examples of what can be done with Scapy, both for the next release of the guide and for my own personal knowledge. So if you use Scapy in anger then let me know for what.
So enjoy and please share the love that is Scapy.
Adam
Coming Soon: The very unofficial dummies guide to scapy..
So the last few weeks have been busy for me in terms of throwing myself into learning more about InfoSec, I’ve attended my first BSides event, made some new friends and published my year-long training plan.
One of the outcomes from attending BSides was my declaration that next year I would do a track 3 talk, and I decided that it would be on scapy (not sure why it just seems like a really cool tool). So I have included scapy on my aforementioned training plan and since then I’ve started playing around with it.
Then a week or so ago (time flies by so quick) @balgan tweeted about the lack of a scapy guide, at the time I thought it would be cool if such a thing existed but gave no more thought about it. That is until today, today I decided that I was going to write what shall now be known as “The very unofficial dummies guide to scapy”… no I’m not making up I’ve decided that as an official scapy dummy why not write a guide as I go, that will both enforce what I learn and maybe give back to the InfoSec community in some small way.
Now this guide is not going to be a huge bible of commands and examples, what is it going to be is a concise guide to building packets, seeing the results and providing examples of actual things you can do with it, basically something you can read in a few hours, follow the examples and write some packets.
I am well aware that you can find a lot of scapy related documentation on the internet but although the end result might not be any different to a few hours googling for things, that’s not the point. The point is really the same as this blog, I write about stuff I want to, if you find it useful that’s awesome, if not oh well never mind..
However, if you think this isn’t actually a bad idea and you’ve used scapy for real world things then let me know. If you know of something that scapy is really cool at doing drop me a line and I will include it in the guide. Remember this is a guide for the community so why not contribute if so inclined..
Environment Disclosure via #shodan
First of a big thanks to @achillean and his awesome website over at http://www.shodanhq.com, the amount of information that gets collected and stored is mind-blowing. I had a brief email conversation with John when I decided to write this blog and at the time there were over 70 million records stored in ShodanHQ.
So to the point of this blog post, in my current job I work a lot on e-commerce type stuff, mostly because I’m responsible for the load balancers we use (if you’ve read this blog before you might be able to guess what they are..). Part of that work means every now and again I get sent the output of our regular pen tests to answer questions or fix “holes”.
One of the most common “holes” I fix is what the external pen testers call “Environment Disclosure Information“, which in layman’s terms means you are giving out more information that you should to external people when they visit your websites.
This is an example HTTP header extract from a website, which will highlight the sort of stuff I mean:
Access-Control-Allow-Origin: *
Cache-Control: no-cache, no-store, max-age=0, no-transform, private
Content-Length: 43
Content-Type: image/gif
Date: Sun, 13 May 2012 10:35:11 GMT
Etag: “4FAF8E5F-48B6-0D239661″
Expires: Sat, 12 May 2012 10:35:11 GMT
Last-Modified: Mon, 14 May 2012 10:35:11 GMT
Pragma: no-cache
Server: Omniture DC/2.0.0
Vary: *
X-C: ms-4.4.5
p3p: policyref=”/w3c/p3p.xml”, CP=”NOI DSP COR NID PSA OUR IND COM NAV STA”
xserver: www4
Now remember I’m no security expert but to me this amount of “free” information about your web environment is both unnecessary and well to be fair a bit sloppy.
Looking at the HTTP header above an unethical type of person can determine the type of server you are running (Server: Omniture DC/2.0.0) and the version its running. Which would make it easier when looking for known vulnerabilities, and you can tell that they have at least 4 web servers (xserver: www4) providing this content (which means some sort of load balancing).
This is another HTTP header from a rather “large” software company that like Marmite you either love or hate..
Cache-Control: max-age=0
Connection: close
Content-Length: 12941
Content-Type: text/html; charset=utf-8
Date: Sun, 13 May 2012 10:40:24 GMT
Expires: Sun, 13 May 2012 10:39:24 GMT
PPServer: PPV: 30 H: BAYIDSLGN1H57 V: 0
Server: Microsoft-IIS/7.5
Set-Cookie: MSPRequ=lt=1336905624&co=1&id=64855; path=/;version=1
MSPOK=$uuid-b9356970-ea8a-491c-8c62-f367d9460ca3;
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-XSS-Protection: 0
p3p: CP=”DSP CUR OTPi IND OTRi ONL FIN”
Again you will see that the Server: HTTP header is still there, so is this really a security concern? Do pen testers just highlight it as something to put in a report??
Now onto the cool stuff (well it’s cool to me), if you have ever used ShodanHQ you will know that there is an API available, and if you pay a small amount of $$ you can get a lot of functionality. I decided to use that API and write a ruby script that would look through the 70 million records and give me the total number of results that matched some of the most popular HTTP server headers.
This is my code (I have compared the numbers against individual searches with the same server header).
#!/usr/bin/env ruby
require 'rubygems'
require 'shodan'
#Set your Shodan API Key
SHODAN_API_KEY = "enteryourapihere"
#Create the API object
api = Shodan::WebAPI.new(SHODAN_API_KEY)
#Define the array of Server headers you want to search for
array = ["Apache/2.4","Apache/2.3","Apache/2.2.21", "Apache/2.2.20", "Apache/2.2.19", "Apache/2.2.18", "Apache/2.2.17", "Apache/2.2.16", "Apache/2.2.15", "Apache/2.2.14", "Apache/2.2.13", "Apache/2.2.12", "Apache/2.2.11", "Apache/2.2.10", "Apache/2.2.9", "Apache/2.2.8", "Apache/2.2.6", "Apache/2.2.5", "Apache/2.2.4", "Apache/2.2.3", "Apache/2.2.2", "Apache/2.2.0", "Microsoft-IIS/7.5", "Microsoft-IIS/7.0", "Microsoft-IIS/6.0", "Microsoft-IIS/5.0", "Microsoft-IIS/4.0", "Microsoft-IIS/3.0", "Microsoft-IIS/2.0", "Microsoft-IIS/1.0", "nginx", "squid", "lighttpd"]
begin
#For each value in array, search through Shodan
array.each_index {|s| d = api.search("#{array[s]}")
#Print the array value and the total number of matches against the array value
puts "#{array[s]}: #{d['total']}"}
end
I know it’s nothing flash, but it works..
Now the results (drum roll please)…Bear in mind this isn’t all the web server versions, just the ones I could think of or find without spending hours crawling through the internet.
Results:
Apache/2.4: 465
Apache/2.3: 531
Apache/2.2.21: 229250
Apache/2.2.20: 72756
Apache/2.2.19: 72666
Apache/2.2.18: 4048
Apache/2.2.17: 351696
Apache/2.2.16: 444607
Apache/2.2.15: 328945
Apache/2.2.14: 517311
Apache/2.2.13: 141590
Apache/2.2.12: 81345
Apache/2.2.11: 346329
Apache/2.2.10: 89642
Apache/2.2.9: 743891
Apache/2.2.8: 420166
Apache/2.2.6: 97186
Apache/2.2.5: 63
Apache/2.2.4: 131883
Apache/2.2.3: 2854600
Apache/2.2.2: 28955
Apache/2.2.0: 65168
Microsoft-IIS/7.5: 681421
Microsoft-IIS/7.0: 749303
Microsoft-IIS/6.0: 3932895
Microsoft-IIS/5.0: 506169
Microsoft-IIS/4.0: 14731
Microsoft-IIS/3.0: 603
Microsoft-IIS/2.0: 37
Microsoft-IIS/1.0: 31
nginx: 1299084
squid: 192084
lighttpd: 503577
Yes yes I know, surely someone can’t be using IIS/1.0 but I did triple check that result.. To me that’s a lot of people who either don’t care about hiding this information, or like I said earlier it’s not really a big issue.
So lets take it one step further, ShodanHQ also lets you search the exploitdb using the API. Using the ruby script available from the documentation I ran it against Microsoft IIS/6.0 (the most popular IIS version from my research). Using the script I got 6 “known” exploits back (see below).
Results found: 6
3965: Microsoft IIS 6.0 (/AUX/.aspx) Remote Denial of Service Exploit
8704: Microsoft IIS 6.0 WebDAV Remote Authentication Bypass Vulnerability
8754: Microsoft IIS 6.0 WebDAV Remote Authentication Bypass Exploit (patch)
8765: Microsoft IIS 6.0 WebDAV Remote Authentication Bypass Exploit (php)
8806: Microsoft IIS 6.0 WebDAV Remote Authentication Bypass Exploit (pl)
15167: Microsoft IIS 6.0 ASP Stack Overflow (Stack Exhaustion) Denial of Service (MS10-065)
Now most of these might not be valid because of patching, but out of the 3,932,895 results there might be one or two that hasn’t been patched??
I know that realistically you will never be able to hide everything that might or might not give unethical people an advantage if you become a target, but why make it easy for them??
So is this kind of free information really an issue? If you are pen tester does this kind of information help you when running a test or is it just accepted that it’s out there and available??
Let me know what you think.
What’s in your lab??
So to make things a bit easier as I wander along the path of self enlightenment (or in this case learning more about InfoSec) I thought it was about time I built some sort of “lab” at home, so I can get a better idea of what happens when I say run a nmap scan and to give me something to scan against.
Now it may come as a surprise to you but in the 15 years I’ve worked in IT I’ve never had a server at home.. nope never.. and to be honest I don’t think I need a server now to achieve the results I’m after. Now this is MY lab, its not huge, fancy or flash but it is portable and its low maintenance.
So what did I want from my lab:
1. Simple to maintain
2. Flexibility
3. Performs the tasks I want (always good)
You see some people would (and are entitled to) say that the point of a lab is so you can break things (and learn how to break things) for me, the purpose of my lab was the opposite, well sort of. You see I know what firewall logs say during a port scan, but I don’t know what a port scan looks like in terms of the actual packets sent/received. I’ve got a lot to learn and rather than download a “exploitable” VM and well exploit it I wanted to start at the very beginning.
So my lab setup is very simple.
I have a HP Mini Note 2133 running Security Onion, this is for a mixture of packet captures and IDS alerts. It uses a wireless NIC for the management interface and it’s onboard LAN for the sensor. I have a Checkpoint Safe@Office 500 firewall which will have it’s WAN connection plugged into my home network and I will open ports/services as I need to. Then finally I have my laptop which I will use to either scan the firewall and/or write packets with scapy and run packet captures as I go.
My plan (it’s always good to have a plan) is that to start with the firewall blocking everything, I can review the packet captures and actually see the real responses back (as opposed to the script telling me), when I start working with scapy I can write custom packets and see what effect that has. Then I can slowly start to open ports and compare the results with my initial baseline.
This of course might be the completly wrong way to do things, but to me it makes sense. If I can understand what happens in relation to the packets I hope it will give me a more complete understanding of how things work.
Below is a quick and simple diagram of my lab, written by the way with DroidDia (yes there is a droid version of Dia).
Let me know what you think (if you want) and I will let you know how I get on.
Adam
Linkedin Profile
