So I’ve decided to finally getting around to revising (and then taking) my SANS 503 exam (or GCIA). It’s been a while since I’ve spent anytime looking at packets up close and personal so rather than me suffering alone I thought I would blog as I go (plus it’s a good way to remember stuff).
This is the first in a new series called “Packet Addict” (that’s me by the way).
Today we are going to have a look at IPv4 packets, what makes up a IP packet and how you can read these things in their “naked” version (or Hex if you want to be precise). I’m going to be honest you can find all of this on Google, so go ahead or just carry on reading.
DISCLAIMER: If I get something wrong, please tell me as I’m revising for an exam and you wouldn’t want to feel guilty if I fail because you didn’t tell me something was wrong.. now would YOU.
So lets just cover some key terms before we start.
Bit – Smallest unit of a packet, has the value of 0 or 1 (so either on or off).
Nibble – Made up of 4 bits and has is one hexadecimal value.
Byte – This is 8 bits, or 2 nibbles and is 2 hexadecimal values.
So to put some context (we all love context), a TCP flag (covered later but used here for reference) is made up of 1 bit which means it’s either on or off (set or unset).
An example of a nibble (4 bits) would be the IP version number (currently you either get 4 or 6 but you know).
A byte is something like the IP Time to Live (amount of hops before the packet dies a slow and painful death in the darkness that is the internet), working on the basis that it’s 2 hexadecimal values then the maximum it can be (or should be) is 255 (0xFF).
To convert hex to decimal in python you can use this snippet of code:
>>> print int("5", 16)
Hexadecimal is a base 16 numbering system, so the python code takes your value (5 in this case) and prints the int value based on it being base 16.
All make sense?? Awesome lets carry on.
So an IPv4 packet is a minimum of 20 bytes with a maximum of 60 bytes, most IPv4 packets are 20 bytes and I’ve not come across one that is larger than that but I’ve led a settled life.
So in order to work out from hex (that’s hexadecimal in big word speak) what a IPv4 header is really saying (without using Wireshark) you need one of these things below (not mine, found it on the internet)..
This is a graphical representation of an IPv4 header and it’s associated bits (and bytes), so now with the magic of a calculator and Google for when I get stuck, lets make some packet magic.
OK so here is the hex dump of a IPv4 header (taken from a random pcap file).
4500 003c 2d70 4000 4006 d7f6 0a14 9096 0a14 9097
So lets work through the first few bits together. Every IPv4 packet starts (typically) with 4500 which makes it a bit easier to spot when looking at lots of raw packets. So remembering that each hex character is a nibble (2 bits) and using the IPv4 Header image from above we can begin to break out the individual parts of the packet.
Version -> 4 bits (1 hex) = 4
IHL -> 4 bits (1 hex) = 5
Now this is the bit that confuses me, how does a 5 in hex equal 20 bytes?? (the standard length of an IPv4 header)… Google to the rescue..
The IHL (Internet Header Length) field is 4 bits long and specifies the header length in 32-bit word (4 bytes).
So there are two ways to do this, my way and the right way but mine is quicker. 4 bits for IHL, multiple that by your IHL value (5) and you get 20 bytes.
The correct way is to do it like this:
5 * 32 / 8 = 20 (bytes)
5 is the IHL value, multiple by 32 (32 bit word), divide by 8 (number of bits in a byte).
So confused yet?? I know I am (trying writing a blog about this stuff), but lets continue. Next we will break down each chunk of the packet and then we can convert it back to decimal (i.e. readable format) later. Using the table above and the raw packet I will map each value into the diagram below with the hex values.
So I’ve tried to match the colouring coding across the original IPv4 Header picture a bit further up. Lets work through each value, convert it to decimal and then we should be able to see it make a bit more sense.
IP Version = 4 – Makes sense it’s an IPv4 packet after all
IHL = 5 – So we know that we need to multiple this by 4 to get the number of bytes = 20 bytes
Type of Service = 00 – So this is a result of zero, not uncommon to see in IP headers
Total Length = 003c – If we convert this to decimal we get 60, which is the size of the packet in bytes
IP Identification Number = 2d70 – This converts to 11632, this value is usually used for identifing the group of fragments of a single IP datagram
Flags = 4 – These flags are used for determining if a packet is fragmented or not and each of the 3 flags is a bit value (either 1 or a 0). The flags are:
bit 0: Reserved, must be zero
bit 1: Don’t Fragment (DF)
bit 2: More Fragments (MF)
So if we look at this from a binary point of view (as we are dealing with bits), we would get 0 1 0 0 which is hexadecimal for 4 (Don’t Fragment). If it was a fragmented packet it would be binary of 0 0 1 0 which is hexadecimal of 2.
Fragment Offset = 000 – A value of zero means it’s either the first fragment, but in this case because the Don’t Fragment bit isn’t set then there is no need for an offset.
Time to Live = 40 – This converts back to 64 decimal which is a “standard” TTL for a packet
Protocol = 06 – This converts to 6 decimal which is the protocol number for a TCP packet (the next layer)
Header Checksum = d7f6 - These are used for error checking, this value converts to 55286
Source IP = 0a149096 – So we know that the maximum value for an IPv4 Address is 255, so each octet of the IP address will be 2 hex values seperated by a “.”, so
if we break each down in 2 hex value pairs we get 0a 14 90 96, which converts to 10.20.144.150.
Destination IP = 0a149097 – Same principal as before, we turn this value into 0a 14 90 97 which in turns converts to 10.20.144.151
So there you go, a fully decoded IPv4 packet.. trust me it was as painful to write as it was for you to read.. :)
More information about the protocol numbers can be found here:
and if you want to see more details about the IPv4 structure you can read this:
So a while back I blogged about the future of sniffMyPackets and how I was looking at building components for it that would make better use of existing systems you may be using (not over tooling things). Gobbler is the first step of that journey and is now available for your amusement..
Gobbler can be found HERE:
Essentially you feed Gobbler a pcap file and tell it how you want the data outputted. The current supported outputs are:
1. Splunk (via TCP)
2. Splunk (via UDP)
3. JSON (just prints to screen at the moment).
The tool is based on Scapy (as always) and once the initial file read is done it’s actually quite quick (well I think it is).
Moving forward I am going to add dumping into databases (mongodb to start with) and Elastic search. On top of this I’m going to start writing protocol parsers for Scapy that aren’t “available”. I’ve found an HTTP one written by @steevebarbeau that is included in Gobbler so you can now get better Scapy/HTTP dumps.
To use with Splunk is easy, first in your Splunk web console create a new TCP or UDP listener (see screenshots below).
NOTE: UDP Listeners are better for any pcap with over 10,000 packets. I’ve had issues with TCP listeners past that point.
Once your Splunk Listener is ready, just “tweak” the gobbler.conf file to meet your needs:
server = ‘localhost’
port = 10000
protocol = ‘udp’
If you created a TCP listener, change the protocol to ‘tcp’ and gobbler will work out the rest for you.
To run gobbler is easy (as it should be), from the gobbler directory run this command:
./gobbler.py -p [pcapfile] -u [upload type]
I’ve included a 1 packet pcap in the repo so you can straight away test using:
./gobbler.py -p pcaps/test.pcap -u splunk
or if you want to see the results straight away try the json option:
./gobbler.py - pcaps/test.pcap -u json
If you load the packets into Splunk, then they will look something like this:
This means if you were to search Splunk for any packet from a Source IP (of your choice) you can just use (test_index is where I am storing them):
This means you can then use the built-in GeoIP lookup to find out the City etc using this Splunk Search:
index="test_index" | iplocation ip_src
Which would give you back the additional fields so you can do stuff like this:
The benefit of using Gobbler is that I’ve done all the hard work on the formatting, I was going to write a Splunk app but the people at Splunk have been ignoring me so…
I’ve successfully managed to import about 40,000 packets via a UDP listening in just a few minutes, it’s not lightning quick but it’s a start.
PS. A “rebranding” of my core projects will be underway soon so stay tuned for more updates.. :)
So about a year ago I started work on “sniffMyPackets”, the Maltego transform set (using Canari Framework) for analysing pcap files. I started it for 3 reasons;
1. I’m obsessed with pcap files (I admit it, I’m an addict)
2. I wanted to start writing Python code
3. It sounded like fun
At the end of last year I hit a wall, 68 transforms in with what I think is a pretty cool “flow” to it and I ran out of ideas, direction, steam, whatever you want to call it. So I haven’t really done much with it since.
Then over the last few months, with the Scapy workshop at BSides and a mental kick up the ass I decided that I should give it some love (virtual of cause). The next question was “OK so how do I make it better??”, I had an idea, I thought it was awesome that was going to be the next big release for SmP (sniffMyPackets), but as the weeks have gone on I’m starting to wonder if I”m looking at this wrong. What benefit would this give the people that use it??
Let me try and explain… (bear with me)
People write tools to help other people, to enable them to do the same task over and over again with little effort and to ensure the results are predictable (within a small margin of error). So I had written a tool (SmP), within another tool (Maltego), using another tool (Canari Framework) to enable me to do that. To be honest when you write it down like that it seems a little crazy. In order to use my tool you need another 2 in place already!!
One of the things that bugs me at work, is when people say “Sorry can’t do that, I need this tool and it costs $$$$”, honestly the amount of times I want to kick them in the nuts and say “Open your eyes, we have loads of tools already, just make the most of them”. The reality is that in IT (especially Operations), we like off the shelf tools, because we’re not developers, we don’t code, we just use the same tools as everyone else does.
So why do I tell you this?? Well so my “awesome” idea for SmP was to start integrating pcaps into a database, with a web front end and then rewrite the Maltego transforms to suck out data from the database. I honestly thought that would be cool. Then while researching I discovered there are lots of “tools” out there already. Things like Splunk, LogStash, ElasticSearch, MongoDB (for databases), so why I am retooling something just for the sake of it (other than learning how to do it)??
So from now on I’m going to write tools that let you integrate into existing toolsets, I want you to use my code to get more out of what you already have rather than forcing you to get another toolset to manage and support. You want pcap files in Splunk??? Right I’m going to write you a App for that, you want to use ElasticSearch?? Awesome I’ll do that as well. You want pcap files in json? Easy..
The existing Maltego set for SmP will stay (as I have at least 1 user) but I will rewrite them to plug into existing toolset and suck the same data out, that way you don’t need more tools, you can just get more of the ones you’ve already committed to.
PS sorry for any spelling/grammar issues, I’m not wearing my glasses..
So while doing my Scapy Workshop at BSides London the other week, I stated that iterating over DNS response records with Scapy is a bit of a ball ache. Well I will be honest, I was kind of wrong. It’s not that difficult it’s just not that pretty.
This is an example of a DNS response (Answer) packet when running a dig against http://www.google.com
You will see that there are 5 DNSRR layers in the packet, now when you ask Scapy to return the rdata for those layers you will only get the first one (in the Scapy code below pkts refers to the second packet in the pcap which is the response packet).
In order to get the rest, you need to iterate over the additional layers.
In the example above the  and  are the layer “numbers” so to make that a bit easier to understand.
pkts = Ether
pkts = IP
pkts = UDP
pkts = DNS
pkts = DNSQR
pkts[5-9] = DNSRR
So the code below is a quick way to iterate over the DNS responses by using the ancount field to determine the number of responses and then working backwards through the layers to show all the values.
from scapy.all import *
pcap = 'dns.pcap'
pkts = rdpcap(pcap)
for p in pkts:
a_count = p[DNS].ancount
i = a_count + 4
while i > 4:
print p[i].rdata, p[i].rrname
i -= 1
So there you go, quick and dirty Scapy/Python code.
So this week (Tuesday) was the 4th annual BSides London event held at the Kensington and Chelsea Town Hall (same venue as last year). For the last 3 years I’ve attended the event as not only a participant but also as a crew member, helping make the event awesome (which it is every year) and making it a tradition not to see ANY of the talks.. For me BSides is more about taking part and just meeting loads of cool people rather than going to all the talks and snagging all the free stuff, well ok apart from the MWR t-shirts but they are awesome.
This year however was slightly different, a new twist on an already awesome day. Leading up to the event I was busy helping (I use the word loosely) keep the website up-to-date (oh how I hate HTML) and generally just counting down the days.
This year I had planned on attending one workshop on a subject very close to my heart.. Scapy, which was due to be run by Matt Erasmus (@undeadsecurity). However the Thursday before BSides Matt had to pull out which left a 2 hour slot in the BSides schedule free (Can you guess where this is going??).
Friday morning I get an email from Iggy (@GeekChickUK) asking if I fancied running the workshop instead. No my initial panic fuelled response was going to be “God no” but then I thought “Why not, what’s the worse that can happen).
The next 3 days were rammed with me writing a new workshop for a group (well I was hoping at least 1) of people that would most likely either professional infosec ninja’s (they are all ninja’s right??) or at least be able to point out my mistakes when I made them.
On the day I think about 18 people attended my workshop, most of them laughed at my jokes and most of them (hopefully) learnt something new about how awesome Scapy is. I’ve just found out the online feedback form for the whole BSides London event contains questions about my workshop so I will leave judging it’s success till I see those..
If nothing else it’s made me want to run more workshops, not just on Scapy but on other areas as I learn them, so I just want to say a BIG THANK YOU to Iggy for giving me that nudge and of course all the people that attended my workshop on the day.
So I’ve just finished pushing my Watcher Project API “live” for you guys to poke around and see what you think. Before I give you the URL here are my standard disclaimers:
1. The API is very basic at the moment and only allows for query by SSID (Access Points).
2. There are currently only 1255 records in the database, all of which are from around my city.
3. The API (and I use the term loosely) is running in the cloud in the smallest possible instance possible so it might be slow, prone to crashing etc. etc.
4. I’m funding this little project so when it gets too expensive to run I will probably have to turn it off.
5. The data isn’t backed up but I have the raw files so..
To be honest even now I still get a sick feeling in my stomach when pushing stuff live, call it fear of negative feedback but luckily I tend not to get any feedback so yah me.. :)
All of the access points have been imported from Kismet .netxml files and I’ve been using Kismet running on an old Nokia N900 pwnphone which works well and has the benefit of having GPS. So if you have any Kismet files you want to share, let me know..
You can access the API over your browser or just by requesting a normal GET request over HTTP to the server. You will get in a return a JSON response similar to this one below.
“enc”: “None “,
“datetime”: “Thu Mar 20 20:30:50 2014″
Over the next few weeks I will be adding some extra functionality, things like:
1. SSL Certificate
2. More search functions
3. More access points
4. Web interface for searching
5. Some sort of backup (mongodump to S3 bucket or something)
6. Stop it being case sensitive on queries
7. File upload and import for users
8. Some kind of token auth
Most of this is a learning experience for me rather than anything else, I will also add some transform into my Watcher Maltego pack to query the API so it that all ties in nicely.
So here is the URL (see if anyone spots the port number choice).
You then add your SSID to the end, for example.
Here are some SSID’s in the database to get you testing.
McDonald’s Free WiFi
Please, please, please let me know what you think and if it gets better/bigger its useful.
Hello reader(s), I hope you are well and enjoying the onset of spring (here in the UK that means rain.. and lots of it)..
So a few weeks ago I released “Project Watcher” which were my wireless transform for Maltego. It was a bare bones release intended to get them out there and see what people thought. I’m pleased to say I’ve received no constructive feedback and as such I’m following the same motto as always.. “Sod you, I’m having fun ..”
While I was writing the first transforms one of the things that I found was it there weren’t any open source Wardriving databases with a nice HTTP based API, so I thought as the next stage of Project Watcher I would create one..
Today I finished the prototype of a what I hope soon to release to the public. Basically it will allow people to use a simple HTTP GET request to query the Watcher database and see if a wireless access point has been collected and stored. This is a “no frills” solution, there isn’t a web page to look at just an API (I’m not going to call it a RESTful API because it’s not, well not yet).
This was all new to me as I don’t code (other than python) but the API will use MongoDB and a Node.js front end to allow people to query the database. To be honest if it doesn’t work I will just turn it off, I’m running this at my own expense so it’s more about learning new things but if it takes off that would be awesome.
I’ve written some python that allows me to take the watcher.db (sqlite) database and import into MongoDB and my next piece of work is to write some code be able to import kismet files into the database via a web page (so you guys can have a go).
Longer term I hope to get a web site up to allow people to search and a couple of other bits that I’m keeping secret… :)
So stay turned for updates the next few weeks..