Category Archives: General

Project Watcher – The next phase..

Hello reader(s), I hope you are well and enjoying the onset of spring (here in the UK that means rain.. and lots of it)..

So a few weeks ago I released “Project Watcher” which were my wireless transform for Maltego. It was a bare bones release intended to get them out there and see what people thought. I’m pleased to say I’ve received no constructive feedback and as such I’m following the same motto as always.. “Sod you, I’m having fun ..”

While I was writing the first transforms one of the things that I found was it there weren’t any open source Wardriving databases with a nice HTTP based API, so I thought as the next stage of Project Watcher I would create one..

Today I finished the prototype of a what I hope soon to release to the public. Basically it will allow people to use a simple HTTP GET request to query the Watcher database and see if a wireless access point has been collected and stored. This is a “no frills” solution, there isn’t a web page to look at just an API (I’m not going to call it a RESTful API because it’s not, well not yet).

This was all new to me as I don’t code (other than python) but the API will use MongoDB and a Node.js front end to allow people to query the database. To be honest if it doesn’t work I will just turn it off, I’m running this at my own expense so it’s more about learning new things but if it takes off that would be awesome.

I’ve written some python that allows me to take the watcher.db (sqlite) database and import into MongoDB and my next piece of work is to write some code be able to import kismet files into the database via a web page (so you guys can have a go).

Longer term I hope to get a web site up to allow people to search and a couple of other bits that I’m keeping secret… :)

So stay turned for updates the next few weeks..

2014 – Change is around the corner

Hello readers, I hope you are well and this blog post finds you all in good health and excellent spirits.. Well enough about you, this is my blog after all so on to me.. :)

The last few months have been challenging, my initial high of InfoSec learning and drive has seemed to dropped and instead I’ve been left with a sense of emptiness in terms of what and where to go next. If you remember I started this journey nearly 2 years ago with the sole purpose of doing more “security stuff” and overall I have to say I’ve achieved my goal. He’s a quick recap of what I’ve done (yeah I know I’m blowing my own trumpet but lets face it, if you could, you would).

  • OSCP – Done
  • OSWP – Done
  • Malware course – Done
  • SANS course – Done
  • Wrote some cool code (well I think it’s cool) – Done
  • Wrote the “Very Unofficial Dummies Guide to Scapy” – Done
  • Met some really cool people and even got to see a bit more of the world out of it – Done

So where to go from here?? A few people who I have a great deal of respect and time for suggested that instead of my scatter gun approach to learning I focus more on one or two areas, which to be fair makes perfect sense. The problem is on what, I needed to understand my “bliss”, the thing that you love the most and are passionate about. You know that thing that can consume hours of your time without you even realising (no not Christmas shopping).

It’s taken me weeks to work out what my “bliss” is, and in the end it turned out to be quite simple. Throughout my career I’ve built things, designed things, devised solutions to problems that other people have struggled with. One of my greatest assets is my imagination, my desire to learn new things and to push the boundaries of “the norm”. It’s what I enjoy, it’s my bliss.

So what does this mean, I hear you ask. Well throughout 2014 I’m going to take the 16 years infrastructure knowledge I have and the 2 years of InfoSec skills I’ve developed to build things. I have no idea what yet but with my new (and oddly strange) love for coding it’s more likely to be taking an idea that randomly pops into my head (very random at times) and turning it into something, always with a security twist. I want to see what focusing on creating things can lead to. I’ve already experienced it with my sniffMyPackets work, and I want to see what else I can do.

For me, that’s the true meaning of “hacker”, not these Hollywood hackers that take down systems with a single keystroke but someone who builds something, that can take an idea and make something out of it (whether it’s a bad idea or not), or takes an idea from someone else (giving full credit to the original creator) and tweaking it for new and interesting mischief.

I already have a few ideas locked away in the attic that is my brain and it’s time to dust off my IDE and start making things go boom (not really boom if you are reading this Mr NSA).

So if I don’t get a chance before, I wish you all a very merry Christmas/New Year etc etc. and may you all find your bliss in 2014.

Assembly – Resources

Hello reader(s), hope all is well in your world.

So at the moment I’m working through Coursera’s “Malicious Software and its Underground Economy: Two Sides to Every Story” 6 week course. It’s actually quite good and I’m learning new things as well as reinforcing things that are stuck in my head somewhere.

The second lecture is all about Malware Static Analysis and they give you a quick overview on Assembly, which needless to say made my head hurt. So I reached out to my Twitter followers and asked for recommendations of books/videos/websites that will provide a “dummies intro to assembly”.

Like the awesome community this is they provided me with some awesome suggestions/links so I thought I would share:

Free Online Training:



Enjoy and let me know if I’ve missed any hidden gems along the way..

Rant: Community is King

So I don’t think I’ve ever done a rant blog post, and to be fair there is no real reason behind this I just started thinking about it on the way into work (which is about a 10 minute drive). Shall we begin??

DISCLAIMER: I apologise in advance for any bad language used during this rant or the excessive use of “”.

A couple of weeks ago I had reason to tell someone (over email) a little about myself in an attempt to “sell” myself. It’s not something I like doing but sometimes you just have to. It made me realise that during the last 18 months that I’ve been “trying to get into Security” that I’ve actually achieved a lot so I hope this rant will help people who are in the same situation as me.

Community is King???

This time 2 years ago I would spend most of my downtime playing computer games, call it a lack of motivation, laziness or whatever but that’s what I did, then with some gentle pushing from my nearest and dearest I decided to start using my time to learn and develop. When you start with the goal of “breaking into Security” many people point out that the key to success is “the community” and it’s true but that can be the hardest challenge. If you don’t work in Security then some people will tell you it’s just a hobby and maybe they are right or maybe that’s just bollocks, it’s for you to decide and ultimately turn it into anything you want.

I’ll let you into a secret, I started this blog for 2 reasons, the first was to keep a record of what I’ve done and allow me to pat myself on the back for the number of visitors I get, the second was because I wanted to get noticed, I hoped that over time people would read my blog, follow me on twitter and allow me into their circle of InfoSec friends and maybe if I was lucky I might end up with a job out of it. Then I realised something, and some people might disagree but its my blog not yours..

“You don’t have to work in Security, to be in Security”

Not really groundbreaking is it but it’s important because well it’s the point of this post. Over the last 18 months I’ve done a fair few bits and pieces for “the community” I’ve met some awesome people, done some awesome things and have even more awesome things on the horizon and 98% of that was from the community. If people tell me Security is just my hobby my first reaction is to tell them to “do one” because I have hobbies and they don’t consume the amount of time I put into projects, blogging, helping with events. Hobbies don’t consume your time like this does, they don’t push you to go further, learn more, make yourself better and give you that feeling that you can make a difference. This isn’t a hobby, it’s not my career either but doesn’t make it any less, its part of who I am and always will be.

So if you are just starting in Security and find yourself a little unmotivated because you can’t find that dream Security job or you are finding the community a bit “cliquey” here are my top tips:

1. Write it and they will come – Remember that awesome blog post (not this one) you read about the latest exploitation technique? Or that tool you used? Someone took the time to write that and then out of the goodness of their heart gave it away for free to YOU. Don’t you think it would be nice to repay the favour?? Seriously if you just start writing code, making videos or writing articles people will find them, share them and slowly over time you will find yourself more involved in the community than you ever expected.

2. Twitter isn’t just about your latest bowel movement – Follow people on twitter, it’s a good way to find people who post all that useful stuff you read. Interact with them by all means but remember this.

To start with they will probably ignore you, won’t follow you and generally see you as noise on their timelines, but give it time and slowly you will get there. I get more followers from Twitter from blog posts/code release than just by talking to people, and just accept that some people are very picky about following back or even replying back if you mention them in Tweets.

3. You’re never alone – In the UK there aren’t a lot of conferences, CTF events and only limited events, if there isn’t anything in your area then start something, you want to be part of the community then sometimes you have to make it happen. If you want to organise a monthly Security focused meeting in your area then do it, don’t let people tell you can’t, because well you can. Even if only 1 other person turns up that’s 1 person you didn’t know who shares the same interests as you (unless it’s your mum).

4. It’s up to you – If you want to make Security just a hobby, then that’s fine. If you want to make it a career that’s awesome but it’s up to you to decide and more importantly it’s up to you to make it happen. Don’t let other people label what your passions, dreams or ambitions are, they are yours and no one elses.

OK that’s the rant over with. Thanks for listening.

Code: PDF hunter

So of late I’ve been playing around a lot with Scapy and pcap files, mostly for my sniffMyPackets project but also because it teaches me more about network forensics and python. The other area I’m starting to learn about is Malware Analysis and I’ve been spending some time looking at the Honeynet Project challenges.

One of the challenges to is to find the malicious content within a PDF file that is provided to you in a pcap file. Normally I would just reach for Network Miner and rebuild the file(s) that way but I wanted to see if I could write some code myself.

The goal of my code was simple, parse through a pcap file, identify a PDF and then rebuild the file so that if a tool such as exiftool or file was used that it would correctly be identified as a PDF and that you could open the PDF and view the content (if you wanted to).

I follow a certain process when I’m carving up pcap files, it’s not rocket science really just common sense. First off find the packets you are interested in, I tend to use a mix of Wireshark and Scapy for this and then look for something you can use to filter down to the packets you want before getting into the nitty gritty of carving them up.

For this piece of code I need to find some way of identifying a PDF file in a pcap file and as most PDF files will appear in a pcap file as part of an HTTP conversation, I parse each packet and if the packet has a Raw layer (a raw layer in Scapy is essentially the payload of a packet) then I look for this ‘Content-Type: application/pdf’. If this is matched then I store the TCP ACK number as a variable for use later.

Now once I have the ACK number I then need to find all the packets that relate to this in order to get the whole file. Now it turns out the ACK is the same for all the packets that the PDF download is in (something I didn’t realise until I started this) so it’s a simple case of using the following code to find all the packets I’m after:

for p in pkts:
if p.haslayer(TCP) and p.haslayer(Raw) and (p.getlayer(TCP).ack == int(ack) or p.getlayer(TCP).seq == int(ack)):
raw = p.getlayer(Raw).load

If either the TCP ACK or SEQ match our stored ACK variable we get the Raw layer and store it into a python list. This means that we now have (hopefully) all the packets that make up the PDF stored nicely away and because it’s a TCP conversation they should all be in the right order.

Now that we have all the packets we write those out to a temporary file, it’s a temporary file because if you were to open it in a text editor you would see all the HTTP headers at the top and the bottom, which means if you ran file against it, then you would get back a file type of “data” and not “PDF” (which is what we are after).

So we then have to do some python magic (well I think it’s magic), to slice the rubbish out. Now this is the part that took me the longest to figure out. If you have ever looked at a PDF file in a text editor (I wouldn’t blame you if you haven’t), you would notice that they start with “%PDF-“ and end with “%%EOF” so finding the start of a PDF file is easy, the problem is that a PDF file can have multiple %%EOF towards the end of the file and I kept cutting at the wrong point.

To fix this I came up with a bit of a long-winded way of carving the temporary file up (see the code below):

# Open the temp file, cut the HTTP headers out and then save it again as a PDF
total_lines = ''
firstcut = ''
secondcut = ''
final_cut = ''

f = open(tmpfile, 'r').readlines()

total_lines = len(f)

for x, line in enumerate(f):
if start in line:
firstcut = int(x)

for y, line in enumerate(f):
if end in line:
secondcut = int(y) + 1

f = f[firstcut:]

if int(total_lines) - int(secondcut) != 0:
final_cut = int(total_lines) - int(secondcut)
f = f[:-final_cut]

If you read Python awesome, if you don’t here’s what happens.

First off I open the temporary file and count the number of lines, I look for the variable I declared at the start of the code as start (which is this: start = str(‘%PDF-‘)), if that’s matched it stores the line number as the variable firstcut

I then need to find the last cut, I look for the variable end (which is this: end = str(‘%%EOF’)) now remember I said a PDF can have multiple EOF statements, well I get round that because Python overrides the variable secondcut each time it’s matched so the last line with EOF is always the one used. I also add a +1 to the line number because for the next chunk of code if I didn’t I would actually cut the final %%EOF file the file (I know this because I did it, before realising what was happening).

So we now do a simple little IF statement to make sure that there is something at the end of the file to cut (sometimes there isn’t on the pcap files I’ve used/made) and if there is we slice the bad HTTP headers out before saving the file. If there isn’t anything to cut then we just save the file.

Hopefully that makes sense to non-python people (I can but hope).

I’ve tested this on a number of different pcap files that have PDF downloads in them and it works, I can open and view the PDF and if I run file or exitfool against it then it appears as a normal PDF. I’m sure there are some cases when it won’t work 100% but if you find something that doesn’t let me know so I can try to fix it.

The code can be found here: (in my ever-growing GitHub repo). Oh and I’ve added this function into my sniffMyPackets transform pack.


Coming Soon: The very unofficial dummies guide to scapy..

So the last few weeks have been busy for me in terms of throwing myself into learning more about InfoSec, I’ve attended my first BSides event, made some new friends and published my year-long training plan.

One of the outcomes from attending BSides was my declaration that next year I would do a track 3 talk, and I decided that it would be on scapy (not sure why it just seems like a really cool tool). So I have included scapy on my aforementioned training plan and since then I’ve started playing around with it.

Then a week or so ago (time flies by so quick) @balgan tweeted about the lack of a scapy guide, at the time I thought it would be cool if such a thing existed but gave no more thought about it. That is until today, today I decided that I was going to write what shall now be known as “The very unofficial dummies guide to scapy”… no I’m not making up I’ve decided that as an official scapy dummy why not write a guide as I go, that will both enforce what I learn and maybe give back to the InfoSec community in some small way.

Now this guide is not going to be a huge bible of commands and examples, what is it going to be is a concise guide to building packets, seeing the results and providing examples of actual things you can do with it, basically something you can read in a few hours, follow the examples and write some packets.

I am well aware that you can find a lot of scapy related documentation on the internet but although the end result might not be any different to a few hours googling for things, that’s not the point. The point is really the same as this blog, I write about stuff I want to, if you find it useful that’s awesome, if not oh well never mind.. :)

However, if you think this isn’t actually a bad idea and you’ve used scapy for real world things then let me know. If you know of something that scapy is really cool at doing drop me a line and I will include it in the guide. Remember this is a guide for the community so why not contribute if so inclined..

Environment Disclosure via #shodan

First of a big thanks to @achillean and his awesome website over at, the amount of information that gets collected and stored is mind-blowing. I had a brief email conversation with John when I decided to write this blog and at the time there were over 70 million records stored in ShodanHQ.

So to the point of this blog post, in my current job I work a lot on e-commerce type stuff, mostly because I’m responsible for the load balancers we use (if you’ve read this blog before you might be able to guess what they are..). Part of that work means every now and again I get sent the output of our regular pen tests to answer questions or fix “holes”.

One of the most common “holes” I fix is what the external pen testers call “Environment Disclosure Information“, which in layman’s terms means you are giving out more information that you should to external people when they visit your websites.

This is an example HTTP header extract from a website, which will highlight the sort of stuff I mean:

Access-Control-Allow-Origin: *
Cache-Control: no-cache, no-store, max-age=0, no-transform, private
Content-Length: 43
Content-Type: image/gif
Date: Sun, 13 May 2012 10:35:11 GMT
Etag: “4FAF8E5F-48B6-0D239661″
Expires: Sat, 12 May 2012 10:35:11 GMT
Last-Modified: Mon, 14 May 2012 10:35:11 GMT
Pragma: no-cache
Server: Omniture DC/2.0.0
Vary: *
X-C: ms-4.4.5
p3p: policyref=”/w3c/p3p.xml”, CP=”NOI DSP COR NID PSA OUR IND COM NAV STA”
xserver: www4

Now remember I’m no security expert but to me this amount of “free” information about your web environment is both unnecessary and well to be fair a bit sloppy.

Looking at the HTTP header above an unethical type of person can determine the type of server you are running (Server: Omniture DC/2.0.0) and the version its running. Which would make it easier when looking for known vulnerabilities, and you can tell that they have at least 4 web servers (xserver: www4) providing this content (which means some sort of load balancing).

This is another HTTP header from a rather “large” software company that like Marmite you either love or hate..

Cache-Control: max-age=0
Connection: close
Content-Length: 12941
Content-Type: text/html; charset=utf-8
Date: Sun, 13 May 2012 10:40:24 GMT
Expires: Sun, 13 May 2012 10:39:24 GMT
PPServer: PPV: 30 H: BAYIDSLGN1H57 V: 0
Server: Microsoft-IIS/7.5
Set-Cookie: MSPRequ=lt=1336905624&co=1&id=64855; path=/;version=1
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-XSS-Protection: 0

Again you will see that the Server: HTTP header is still there, so is this really a security concern? Do pen testers just highlight it as something to put in a report??

Now onto the cool stuff (well it’s cool to me), if you have ever used ShodanHQ you will know that there is an API available, and if you pay a small amount of $$ you can get a lot of functionality. I decided to use that API and write a ruby script that would look through the 70 million records and give me the total number of results that matched some of the most popular HTTP server headers.

This is my code (I have compared the numbers against individual searches with the same server header).

#!/usr/bin/env ruby
require 'rubygems'
require 'shodan'

#Set your Shodan API Key
SHODAN_API_KEY = "enteryourapihere"

#Create the API object
api =

#Define the array of Server headers you want to search for
array = ["Apache/2.4","Apache/2.3","Apache/2.2.21", "Apache/2.2.20", "Apache/2.2.19", "Apache/2.2.18", "Apache/2.2.17", "Apache/2.2.16", "Apache/2.2.15", "Apache/2.2.14", "Apache/2.2.13", "Apache/2.2.12", "Apache/2.2.11", "Apache/2.2.10", "Apache/2.2.9", "Apache/2.2.8", "Apache/2.2.6", "Apache/2.2.5", "Apache/2.2.4", "Apache/2.2.3", "Apache/2.2.2", "Apache/2.2.0", "Microsoft-IIS/7.5", "Microsoft-IIS/7.0", "Microsoft-IIS/6.0", "Microsoft-IIS/5.0", "Microsoft-IIS/4.0", "Microsoft-IIS/3.0", "Microsoft-IIS/2.0", "Microsoft-IIS/1.0", "nginx", "squid", "lighttpd"]
#For each value in array, search through Shodan
array.each_index {|s| d ="#{array[s]}")
#Print the array value and the total number of matches against the array value
puts "#{array[s]}: #{d['total']}"}

I know it’s nothing flash, but it works.. :)

Now the results (drum roll please)…Bear in mind this isn’t all the web server versions, just the ones I could think of or find without spending hours crawling through the internet.


Apache/2.4: 465
Apache/2.3: 531
Apache/2.2.21: 229250
Apache/2.2.20: 72756
Apache/2.2.19: 72666
Apache/2.2.18: 4048
Apache/2.2.17: 351696
Apache/2.2.16: 444607
Apache/2.2.15: 328945
Apache/2.2.14: 517311
Apache/2.2.13: 141590
Apache/2.2.12: 81345
Apache/2.2.11: 346329
Apache/2.2.10: 89642
Apache/2.2.9: 743891
Apache/2.2.8: 420166
Apache/2.2.6: 97186
Apache/2.2.5: 63
Apache/2.2.4: 131883
Apache/2.2.3: 2854600
Apache/2.2.2: 28955
Apache/2.2.0: 65168
Microsoft-IIS/7.5: 681421
Microsoft-IIS/7.0: 749303
Microsoft-IIS/6.0: 3932895
Microsoft-IIS/5.0: 506169
Microsoft-IIS/4.0: 14731
Microsoft-IIS/3.0: 603
Microsoft-IIS/2.0: 37
Microsoft-IIS/1.0: 31
nginx: 1299084
squid: 192084
lighttpd: 503577

Yes yes I know, surely someone can’t be using IIS/1.0 but I did triple check that result.. :) To me that’s a lot of people who either don’t care about hiding this information, or like I said earlier it’s not really a big issue.

So lets take it one step further, ShodanHQ also lets you search the exploitdb using the API. Using the ruby script available from the documentation I ran it against Microsoft IIS/6.0 (the most popular IIS version from my research). Using the script I got 6 “known” exploits back (see below).

Results found: 6
3965: Microsoft IIS 6.0 (/AUX/.aspx) Remote Denial of Service Exploit
8704: Microsoft IIS 6.0 WebDAV Remote Authentication Bypass Vulnerability
8754: Microsoft IIS 6.0 WebDAV Remote Authentication Bypass Exploit (patch)
8765: Microsoft IIS 6.0 WebDAV Remote Authentication Bypass Exploit (php)
8806: Microsoft IIS 6.0 WebDAV Remote Authentication Bypass Exploit (pl)
15167: Microsoft IIS 6.0 ASP Stack Overflow (Stack Exhaustion) Denial of Service (MS10-065)

Now most of these might not be valid because of patching, but out of the 3,932,895 results there might be one or two that hasn’t been patched??

I know that realistically you will never be able to hide everything that might or might not give unethical people an advantage if you become a target, but why make it easy for them??

So is this kind of free information really an issue? If you are pen tester does this kind of information help you when running a test or is it just accepted that it’s out there and available??

Let me know what you think.