Maltego – gotFlow (Netflow for Maltego)

Recently I was asked to see if I could create some Maltego transforms to provide a quick analysis of Netflow data. Always up for a challenge (and to feed my Maltego addiction) I created gotFlow, which is based on the Canari Framework (for rapid Maltego transform generation).

gotFlow is designed to support (currently) nfdump and should still be classed as an “early release” (meaning more to come). It’s a nice simple transform set with only 3 transforms, 3 entities and 1 Maltego machine.


The transforms process works as follows:

nfdump file -> source ip -> destination ip -> destination port

The source and destination IP’s are the Maltego IPv4 Address entities allowing you to run additional transforms against them.

To get started you can either add a single nfdump file or import nfdump files from a directory.


From here you can run the ‘[NF] – Import Files’ tranforms that will import all the nfdump files from the chosen directory.

gotFlow-Import Files

Once that’s run you should (depending on the number of nfdump files) get something that looks like this.


You can now either run the Maltego machine against the files or run the transforms seperately. For the purpose of this blog post I’ve cheated and used the machine.


The Machine runs the following transforms, feeding off the return entities generated by the transform before it.

[NF] – Get Source IP
[NF] – Get Destination IP
[NF] – Get Destination Port

What you end up is something like this:


Now I’ve tried to make this a easy to determine traffic type and size by the art of colour coding (very high tech).

TCP Traffic – Red lines
UDP Traffic – Blue lines
ICMP Traffic – Green lines

The thickness of the line between the source IP and destination IP is the size of the flow. The returned value is in bytes which I convert to kilobytes (bytes / 1000). If the line is thin (the default) it means its below 1 kilobyte.

The only configuration change you need to make before you run gotFlow is to define the location of the nfdump executable which needs to be added to:


You can find the transforms here:

Any questions, queries or suggestions let me know (email or raise an Issue on GitHub)

2014 in review

The stats helper monkeys prepared a 2014 annual report for this blog.

Here’s an excerpt:

The concert hall at the Sydney Opera House holds 2,700 people. This blog was viewed about 28,000 times in 2014. If it were a concert at Sydney Opera House, it would take about 10 sold-out performances for that many people to see it.

Click here to see the complete report.

Project Watcher – The next phase..

Hello reader(s), I hope you are well and enjoying the onset of spring (here in the UK that means rain.. and lots of it)..

So a few weeks ago I released “Project Watcher” which were my wireless transform for Maltego. It was a bare bones release intended to get them out there and see what people thought. I’m pleased to say I’ve received no constructive feedback and as such I’m following the same motto as always.. “Sod you, I’m having fun ..”

While I was writing the first transforms one of the things that I found was it there weren’t any open source Wardriving databases with a nice HTTP based API, so I thought as the next stage of Project Watcher I would create one..

Today I finished the prototype of a what I hope soon to release to the public. Basically it will allow people to use a simple HTTP GET request to query the Watcher database and see if a wireless access point has been collected and stored. This is a “no frills” solution, there isn’t a web page to look at just an API (I’m not going to call it a RESTful API because it’s not, well not yet).

This was all new to me as I don’t code (other than python) but the API will use MongoDB and a Node.js front end to allow people to query the database. To be honest if it doesn’t work I will just turn it off, I’m running this at my own expense so it’s more about learning new things but if it takes off that would be awesome.

I’ve written some python that allows me to take the watcher.db (sqlite) database and import into MongoDB and my next piece of work is to write some code be able to import kismet files into the database via a web page (so you guys can have a go).

Longer term I hope to get a web site up to allow people to search and a couple of other bits that I’m keeping secret… :)

So stay turned for updates the next few weeks..

2014 – Change is around the corner

Hello readers, I hope you are well and this blog post finds you all in good health and excellent spirits.. Well enough about you, this is my blog after all so on to me.. :)

The last few months have been challenging, my initial high of InfoSec learning and drive has seemed to dropped and instead I’ve been left with a sense of emptiness in terms of what and where to go next. If you remember I started this journey nearly 2 years ago with the sole purpose of doing more “security stuff” and overall I have to say I’ve achieved my goal. He’s a quick recap of what I’ve done (yeah I know I’m blowing my own trumpet but lets face it, if you could, you would).

  • OSCP – Done
  • OSWP – Done
  • Malware course – Done
  • SANS course – Done
  • Wrote some cool code (well I think it’s cool) – Done
  • Wrote the “Very Unofficial Dummies Guide to Scapy” – Done
  • Met some really cool people and even got to see a bit more of the world out of it – Done

So where to go from here?? A few people who I have a great deal of respect and time for suggested that instead of my scatter gun approach to learning I focus more on one or two areas, which to be fair makes perfect sense. The problem is on what, I needed to understand my “bliss”, the thing that you love the most and are passionate about. You know that thing that can consume hours of your time without you even realising (no not Christmas shopping).

It’s taken me weeks to work out what my “bliss” is, and in the end it turned out to be quite simple. Throughout my career I’ve built things, designed things, devised solutions to problems that other people have struggled with. One of my greatest assets is my imagination, my desire to learn new things and to push the boundaries of “the norm”. It’s what I enjoy, it’s my bliss.

So what does this mean, I hear you ask. Well throughout 2014 I’m going to take the 16 years infrastructure knowledge I have and the 2 years of InfoSec skills I’ve developed to build things. I have no idea what yet but with my new (and oddly strange) love for coding it’s more likely to be taking an idea that randomly pops into my head (very random at times) and turning it into something, always with a security twist. I want to see what focusing on creating things can lead to. I’ve already experienced it with my sniffMyPackets work, and I want to see what else I can do.

For me, that’s the true meaning of “hacker”, not these Hollywood hackers that take down systems with a single keystroke but someone who builds something, that can take an idea and make something out of it (whether it’s a bad idea or not), or takes an idea from someone else (giving full credit to the original creator) and tweaking it for new and interesting mischief.

I already have a few ideas locked away in the attic that is my brain and it’s time to dust off my IDE and start making things go boom (not really boom if you are reading this Mr NSA).

So if I don’t get a chance before, I wish you all a very merry Christmas/New Year etc etc. and may you all find your bliss in 2014.

Assembly – Resources

Hello reader(s), hope all is well in your world.

So at the moment I’m working through Coursera’s “Malicious Software and its Underground Economy: Two Sides to Every Story” 6 week course. It’s actually quite good and I’m learning new things as well as reinforcing things that are stuck in my head somewhere.

The second lecture is all about Malware Static Analysis and they give you a quick overview on Assembly, which needless to say made my head hurt. So I reached out to my Twitter followers and asked for recommendations of books/videos/websites that will provide a “dummies intro to assembly”.

Like the awesome community this is they provided me with some awesome suggestions/links so I thought I would share:

Free Online Training:



Enjoy and let me know if I’ve missed any hidden gems along the way..

Rant: Community is King

So I don’t think I’ve ever done a rant blog post, and to be fair there is no real reason behind this I just started thinking about it on the way into work (which is about a 10 minute drive). Shall we begin??

DISCLAIMER: I apologise in advance for any bad language used during this rant or the excessive use of “”.

A couple of weeks ago I had reason to tell someone (over email) a little about myself in an attempt to “sell” myself. It’s not something I like doing but sometimes you just have to. It made me realise that during the last 18 months that I’ve been “trying to get into Security” that I’ve actually achieved a lot so I hope this rant will help people who are in the same situation as me.

Community is King???

This time 2 years ago I would spend most of my downtime playing computer games, call it a lack of motivation, laziness or whatever but that’s what I did, then with some gentle pushing from my nearest and dearest I decided to start using my time to learn and develop. When you start with the goal of “breaking into Security” many people point out that the key to success is “the community” and it’s true but that can be the hardest challenge. If you don’t work in Security then some people will tell you it’s just a hobby and maybe they are right or maybe that’s just bollocks, it’s for you to decide and ultimately turn it into anything you want.

I’ll let you into a secret, I started this blog for 2 reasons, the first was to keep a record of what I’ve done and allow me to pat myself on the back for the number of visitors I get, the second was because I wanted to get noticed, I hoped that over time people would read my blog, follow me on twitter and allow me into their circle of InfoSec friends and maybe if I was lucky I might end up with a job out of it. Then I realised something, and some people might disagree but its my blog not yours..

“You don’t have to work in Security, to be in Security”

Not really groundbreaking is it but it’s important because well it’s the point of this post. Over the last 18 months I’ve done a fair few bits and pieces for “the community” I’ve met some awesome people, done some awesome things and have even more awesome things on the horizon and 98% of that was from the community. If people tell me Security is just my hobby my first reaction is to tell them to “do one” because I have hobbies and they don’t consume the amount of time I put into projects, blogging, helping with events. Hobbies don’t consume your time like this does, they don’t push you to go further, learn more, make yourself better and give you that feeling that you can make a difference. This isn’t a hobby, it’s not my career either but doesn’t make it any less, its part of who I am and always will be.

So if you are just starting in Security and find yourself a little unmotivated because you can’t find that dream Security job or you are finding the community a bit “cliquey” here are my top tips:

1. Write it and they will come – Remember that awesome blog post (not this one) you read about the latest exploitation technique? Or that tool you used? Someone took the time to write that and then out of the goodness of their heart gave it away for free to YOU. Don’t you think it would be nice to repay the favour?? Seriously if you just start writing code, making videos or writing articles people will find them, share them and slowly over time you will find yourself more involved in the community than you ever expected.

2. Twitter isn’t just about your latest bowel movement – Follow people on twitter, it’s a good way to find people who post all that useful stuff you read. Interact with them by all means but remember this.

To start with they will probably ignore you, won’t follow you and generally see you as noise on their timelines, but give it time and slowly you will get there. I get more followers from Twitter from blog posts/code release than just by talking to people, and just accept that some people are very picky about following back or even replying back if you mention them in Tweets.

3. You’re never alone – In the UK there aren’t a lot of conferences, CTF events and only limited events, if there isn’t anything in your area then start something, you want to be part of the community then sometimes you have to make it happen. If you want to organise a monthly Security focused meeting in your area then do it, don’t let people tell you can’t, because well you can. Even if only 1 other person turns up that’s 1 person you didn’t know who shares the same interests as you (unless it’s your mum).

4. It’s up to you – If you want to make Security just a hobby, then that’s fine. If you want to make it a career that’s awesome but it’s up to you to decide and more importantly it’s up to you to make it happen. Don’t let other people label what your passions, dreams or ambitions are, they are yours and no one elses.

OK that’s the rant over with. Thanks for listening.

Code: PDF hunter

So of late I’ve been playing around a lot with Scapy and pcap files, mostly for my sniffMyPackets project but also because it teaches me more about network forensics and python. The other area I’m starting to learn about is Malware Analysis and I’ve been spending some time looking at the Honeynet Project challenges.

One of the challenges to is to find the malicious content within a PDF file that is provided to you in a pcap file. Normally I would just reach for Network Miner and rebuild the file(s) that way but I wanted to see if I could write some code myself.

The goal of my code was simple, parse through a pcap file, identify a PDF and then rebuild the file so that if a tool such as exiftool or file was used that it would correctly be identified as a PDF and that you could open the PDF and view the content (if you wanted to).

I follow a certain process when I’m carving up pcap files, it’s not rocket science really just common sense. First off find the packets you are interested in, I tend to use a mix of Wireshark and Scapy for this and then look for something you can use to filter down to the packets you want before getting into the nitty gritty of carving them up.

For this piece of code I need to find some way of identifying a PDF file in a pcap file and as most PDF files will appear in a pcap file as part of an HTTP conversation, I parse each packet and if the packet has a Raw layer (a raw layer in Scapy is essentially the payload of a packet) then I look for this ‘Content-Type: application/pdf’. If this is matched then I store the TCP ACK number as a variable for use later.

Now once I have the ACK number I then need to find all the packets that relate to this in order to get the whole file. Now it turns out the ACK is the same for all the packets that the PDF download is in (something I didn’t realise until I started this) so it’s a simple case of using the following code to find all the packets I’m after:

for p in pkts:
if p.haslayer(TCP) and p.haslayer(Raw) and (p.getlayer(TCP).ack == int(ack) or p.getlayer(TCP).seq == int(ack)):
raw = p.getlayer(Raw).load

If either the TCP ACK or SEQ match our stored ACK variable we get the Raw layer and store it into a python list. This means that we now have (hopefully) all the packets that make up the PDF stored nicely away and because it’s a TCP conversation they should all be in the right order.

Now that we have all the packets we write those out to a temporary file, it’s a temporary file because if you were to open it in a text editor you would see all the HTTP headers at the top and the bottom, which means if you ran file against it, then you would get back a file type of “data” and not “PDF” (which is what we are after).

So we then have to do some python magic (well I think it’s magic), to slice the rubbish out. Now this is the part that took me the longest to figure out. If you have ever looked at a PDF file in a text editor (I wouldn’t blame you if you haven’t), you would notice that they start with “%PDF-“ and end with “%%EOF” so finding the start of a PDF file is easy, the problem is that a PDF file can have multiple %%EOF towards the end of the file and I kept cutting at the wrong point.

To fix this I came up with a bit of a long-winded way of carving the temporary file up (see the code below):

# Open the temp file, cut the HTTP headers out and then save it again as a PDF
total_lines = ''
firstcut = ''
secondcut = ''
final_cut = ''

f = open(tmpfile, 'r').readlines()

total_lines = len(f)

for x, line in enumerate(f):
if start in line:
firstcut = int(x)

for y, line in enumerate(f):
if end in line:
secondcut = int(y) + 1

f = f[firstcut:]

if int(total_lines) - int(secondcut) != 0:
final_cut = int(total_lines) - int(secondcut)
f = f[:-final_cut]

If you read Python awesome, if you don’t here’s what happens.

First off I open the temporary file and count the number of lines, I look for the variable I declared at the start of the code as start (which is this: start = str(‘%PDF-‘)), if that’s matched it stores the line number as the variable firstcut

I then need to find the last cut, I look for the variable end (which is this: end = str(‘%%EOF’)) now remember I said a PDF can have multiple EOF statements, well I get round that because Python overrides the variable secondcut each time it’s matched so the last line with EOF is always the one used. I also add a +1 to the line number because for the next chunk of code if I didn’t I would actually cut the final %%EOF file the file (I know this because I did it, before realising what was happening).

So we now do a simple little IF statement to make sure that there is something at the end of the file to cut (sometimes there isn’t on the pcap files I’ve used/made) and if there is we slice the bad HTTP headers out before saving the file. If there isn’t anything to cut then we just save the file.

Hopefully that makes sense to non-python people (I can but hope).

I’ve tested this on a number of different pcap files that have PDF downloads in them and it works, I can open and view the PDF and if I run file or exitfool against it then it appears as a normal PDF. I’m sure there are some cases when it won’t work 100% but if you find something that doesn’t let me know so I can try to fix it.

The code can be found here: (in my ever-growing GitHub repo). Oh and I’ve added this function into my sniffMyPackets transform pack.