Maltego Magic comes to BSides London

I’m a big fan of BSides London, it was the first security conference I ever went to, and this will be my fourth year attending. The last couple of years I’ve been a “crew” member for the event, working in the background to help make the event what we all know and love. Last year I stepped in last-minute to run a Scapy workshop, this year I’ve decided to submit one, on my other favourite thing Maltego.

Below you will find a brief description of the workshop and the things that if you are planning on attending you will need to bring with you.

Maltego Magic – Creating transforms & other stuff

In this workshop I will teach people how to write their own Maltego transforms. Using simple to understand examples (and pictures, everyone likes pictures) I will lead the participants through the process of creating local and remote transforms using just a pen and paper (ok a laptop is needed as well).

A basic knowledge of Maltego & Python is needed but the workshop will be aimed so that anyone can benefit from the magic that is Maltego even if they haven’t coded anything before.

Requirements for the day

  • Laptop (Mac OSX, Windows or Linux)
  • Python installation (2.7 or above, not version 3 though)
  • The Python Requests library (sudo pip install requests)
  • Maltego (CE edition is ok)
  • A text editor that’s Python friendly or a Python IDE (Sublime Text, PyCharm etc)
  • Your imagination (borrow someone else’s if necessary)

The workshop information for the day is below:

Date: June 3rd 2015
Workshop: 3
Track No: 2
Duration: 2 hours
Schedule for: 14:00 – 16:00

If you are interesting in writing Maltego transforms come along to the workshop, if you can’t make it I will be wandering around the con all day so feel free to stop me and we can have a chat about Maltego Magic.

Posted in BSidesLondon, Maltego

Media Monkey – Social Media transforms for Maltego

I’ve spent the last few weeks working on a set of Maltego transforms, the idea being that hopefully they will allow you to query more Social Media sites and get useful information back from them. Now for a change they aren’t local transforms (meaning you have to install them), instead I’ve made them all TDS transforms which makes it easier for you (and more work for me).

Not wanting to break from my silly naming conventions they are named “Media Monkey” slightly inspired by the Chaos & Security Monkey created by Netflix (but in no way associated to them).

Now before you get all excited and starting using them, there are some conditions..

1. Still a work in progress – basically more will be added, some might be changed (or removed).
2. They run on a small server – I’m paying for this so at the moment there isn’t a lot of grunt behind them. Sorry but it’s free so I’m hoping that’s enough for you.
3. All the transforms are over HTTPS and I don’t log anything, unless I turn on debugging for development purposes. IF I do, there is a small chance I might grab a request, but to be honest I’m not interested in what you search for.
4. The most important one.. I am in no way, shape or form responsible for how you chose to use these transforms. Act responsibly (we are all adults) and if you get yourself into trouble its your own fault not mine.

The transforms at the moment cover the following:

GitHub (you need an API key)
Gravatar (surprising how much stuff people put in those profiles)
Amazon Wishlists
BT Phone Book (UK only, still working on it)
BitBucket (this one will likely change soon)

I’ve got loads more to add so the list will change over time.

You can find all the documentation (yes yes I wrote some documentation) HERE

I’m always looking for feedback (good or bad) so if you have ideas, suggestions or even complaints (although I reserve the right to ignore those).

Enjoy!!

Posted in Maltego, OSint

Maltego – gotFlow (Netflow for Maltego)

Recently I was asked to see if I could create some Maltego transforms to provide a quick analysis of Netflow data. Always up for a challenge (and to feed my Maltego addiction) I created gotFlow, which is based on the Canari Framework (for rapid Maltego transform generation).

gotFlow is designed to support (currently) nfdump and should still be classed as an “early release” (meaning more to come). It’s a nice simple transform set with only 3 transforms, 3 entities and 1 Maltego machine.

gotFlow-Entities

The transforms process works as follows:

nfdump file -> source ip -> destination ip -> destination port

The source and destination IP’s are the Maltego IPv4 Address entities allowing you to run additional transforms against them.

To get started you can either add a single nfdump file or import nfdump files from a directory.

gotFlow-Folder

From here you can run the ‘[NF] – Import Files’ tranforms that will import all the nfdump files from the chosen directory.

gotFlow-Import Files

Once that’s run you should (depending on the number of nfdump files) get something that looks like this.

gotFlow-Files

You can now either run the Maltego machine against the files or run the transforms seperately. For the purpose of this blog post I’ve cheated and used the machine.

gotFlow-Machine

The Machine runs the following transforms, feeding off the return entities generated by the transform before it.

[NF] – Get Source IP
[NF] – Get Destination IP
[NF] – Get Destination Port

What you end up is something like this:

gotFlow-Output

Now I’ve tried to make this a easy to determine traffic type and size by the art of colour coding (very high tech).

TCP Traffic – Red lines
UDP Traffic – Blue lines
ICMP Traffic – Green lines

The thickness of the line between the source IP and destination IP is the size of the flow. The returned value is in bytes which I convert to kilobytes (bytes / 1000). If the line is thin (the default) it means its below 1 kilobyte.

The only configuration change you need to make before you run gotFlow is to define the location of the nfdump executable which needs to be added to:

gotFlow/src/gotFlow.conf

You can find the transforms here:

https://github.com/SneakersInc/gotFlow

Any questions, queries or suggestions let me know (email or raise an Issue on GitHub)

Posted in General

sniffMyPackets V2: Database or not??

When I started the work on sniffMyPackets version 2 I decided to make it default, to using a database backend. The decision around this was based on trying to get the most out of the pcap files without crowding the Maltego graph. I knew at the time that this means that people who want to use my code would have to have additional infrastructure for it to run. I’ve tried to minimize the pain by introducing a Vagrant machine that will build a MongoDB database instance and install the new web interface (which is awesome).

Recently I was asked if you “HAD” to use a database and I realised that this decision, might limit the number of people who will use sniffMyPackets. Which that in mind I have now rewritten the Maltego transforms to work with or without a database. The limitation is that don’t get to use all the transforms (such as replaying a session) but you still get the same output on the Maltego graph.

The changes have been pushed to the GitHub repo which is linked below:

https://github.com/SneakersInc/sniffmypacketsv2

By default the database support is switched off, the good news is that it only takes one change to enable it. When you create the Canari profile a sniffmypacketsv2.conf file is created in the src/ directory. So for example:

canari create-profile sniffmypacketsv2 -w /root/localTransforms/sniffmypacketsv2/src

This creates the sniffmypacketsv2.conf file in the /root/localTransforms/sniffmypacketsv2/src directory. The configuration file comes preconfigured with several options most of which you can change to meet your needs. The important one for the database support is under the [working] section and is called ‘usedb’ (see pictures below).

To enable database support, simply change the value from 0 to 1 and you are good to go (so to speak). You can turn it on and off whenever you want.

Database Support Off:

databaseoff

Database Support On:

databaseon

Any Maltego transform that won’t work without the database just returns a message to your Output window about needing database support. To be honest most of the important ones work without database support so you should notice much difference.

If you decide not to use the database then you will be missing out on this cool looking web interface (see below)

coolwebinterface

Over the next few weeks, more Maltego transforms will be created/added and I’ve got some cool features planned in terms of Maltego TDS based transforms and for the web interface as well. I’m also going to start working on adding NetFlow support to sniffMyPackets as well so you can throw even more stuff into the mix.

I will also be working on the documentation and will run a blog series on some of cool features you might not notice straight away.

Oh my online demo site is still online (which sometimes stops working but that’s not my code more the server) and you can find it at:

http://demo.sniffmypackets.net

As always feedback is welcome (good or bad).

Enjoy.

Posted in Maltego, sniffMyPackets | Tagged

sniffMyPackets Version 2 – The Release

If you follow me on Twitter you have probably noticed me bombarding you with tweets about the next release of sniffMyPackets, well today it’s officially released in a Beta format. There is still a lot of work to be done and over the next few weeks/months expect a lot of changes. The purpose of this blog post today is to give you an overview about what this new version is all about, my hope is that you download it, try it and let me know what you think.

DISCLAIMER:

This is a beta, I’ve tested the code as much as possible but you might still have issues, if you do please raise an Issue ticket on the respective GitHub repo.

What’s New:

The original sniffMyPackets was the first piece of code I wrote and it was really more a journey of discovery than an attempt at producing anything that could be classed as “production ready”. This release of sniffMyPackets is my attempt at turning it into something that is not only useful (well hopfully) but also that can be used and shared among analysts within an organisation.

There are two main changes to this release (aside from better Python code), the first is the use of a database backend (currently MongoDB), the second is the introduction of a web interface. These two components work in harmony with the Maltego transforms to give you more information without overwhelming the Maltego graphs but still giving you the visibility you need from within Maltego.

Database:

The database is used by the Maltego transforms to store additional information when transforming a pcap file. When building Maltego graphs you need to be able to tell a story but at the same time not overwhelm the graph with too much information (especially when building large graphs). Each transform when run not only returns a Maltego entity but also writes additional information back into the database. You can then use this information to “replay” a pcap file without needing the actual pcap files (just the Session ID).

For example, when you first add a pcap file into a blank graph you need to index it. This process generates a unique Session ID for that pcap file (based on MD5 hash) which is what is returned to your Maltego graph. What also happens is that additional information is written to the database, which is then displayed in the web interface. Below is the information collected from just that single transform:

  • MD5 Hash
  • SHA1 Hash
  • Comments
  • Upload Time
  • File Name
  • Working Directory
  • Packet Count
  • First Packet
  • Last Packet
  • PCAP ID (or Session ID)
  • File Size
  • While you may think this slows the transform down, I’ve made ever attempt for that not to be the case. A lot of the Python code for this release has been rewritten to speed execution and remove redundant code.

    Web Interface:

    Personally this part has been the greatest challenge for me (but at the same time it’s awesome), the whole purpose behind this addition is to allow you to share the analysis of a pcap file with anyone. As you run transforms from Maltego on a pcap file the web UI is populated with the “detailed” information. Even if you close the Maltego graph and delete the pcap file that information is still there and you can still “replay” that back into Maltego (cool or what..).

    The web UI has some cool features:

    File upload functionality so you can upload pcap files or artifacts from Maltego to the web server to allow for other people to download them.
    Full packet view, when a TCP/UDP stream is indexed in Maltego you can view each individual packet from the web interface (based on Gobbler).

    Other Stuff:

    A lot of the Python code has changed, for example the old method of extracting TCP & UDP streams using tshark has been replaced with pure Python code, this means it’s quicker and removes the need to have tshark/wireshark installed (and stops issues arising when the command syntax changes). The way that sniffMyPackets checks the file types has been turned into a Python function, likewise the way that we check to make sure a pcap file isn’t a pcap-ng format is now a pure Python function.

    I’ve also started to write 3rd party API plugins as well, currently you can submit an artifact MD5 hash to VirusTotal to see if there is an existing report available. In the future the functionality to upload the file to VirusTotal will also be added. I’m also planning on adding CloudShark functionality so you can upload files to a CloudShark appliance or their cloud service.

    Vagrant Machine:

    Now one of my concerns when I decided to add a web interface and a database backend was whether or not people had the infrastructure available to do accomodate this, so in order to try and make it more accessible I created a Vagrant machine. Vagrant is an awesome tool if you do development work, it allows you to script the creation and build of a machine that is repeatable and gives the same results time after time. Within the sniffMyPackets-web repo there is a vagrant folder. If you have Vagrant & Virtualbox installed (required I’m afraid), you can simply type (from within that vagrant folder):

    vagrant up

    This will create a Linux server, download, install and configure the MongoDB database and then the required libraries for the web interface before starting it running. The initial vagrant up can take about 20 minutes (as it has to download stuff) but after that your vagrant up will have your server ready in minutes.

    Maltego Machines:

    There are currently two Maltego Machines available. Both serve a different purpose (kind of obvious I know).

    [SmP] – Decode pcap: This is run after you have indexed a pcap file, it will extract all the TCP/UDP streams, IPv4 Addresses, DNS requests, HTTP requests etc. straight into your Maltego graph.

    [SmP] – Replay Session (auto): If you want to replay a pcap file you can add a Session ID entity (with a valid Session ID), return the pcap file and then run this Maltego Machine against the pcap file. This runs every 30 seconds (useful if you the community edition of Maltego) and will extract all the information in the database for that pcap file into a Maltego graph. This is awesome if you don’t have the original pcap files or what to review information at a later stage that is already in the database.

    How to get it:

    The two GitHub repo’s can be found here:

    Maltego Transforms: https://github.com/SneakersInc/sniffmypacketsv2
    Web Interface: https://github.com/SneakersInc/sniffmypacketsv2-web

    I’m in the process of adding documentation and all that fluffy stuff so bear with me, but if you’ve used sniffMyPackets before then install instructions are the same.

    If you want to see it in action there is YouTube video below:

    Posted in Maltego, sniffMyPackets

    2014 in review

    The WordPress.com stats helper monkeys prepared a 2014 annual report for this blog.

    Here’s an excerpt:

    The concert hall at the Sydney Opera House holds 2,700 people. This blog was viewed about 28,000 times in 2014. If it were a concert at Sydney Opera House, it would take about 10 sold-out performances for that many people to see it.

    Click here to see the complete report.

    Posted in General

    Scapy: Sessions (or Streams)

    I’m in process of rewriting sniffMyPackets version 2 (and yes I’m actually doing it this time), part of work involves doing a complete rewrite of the Scapy based code that I used in the original version. This isn’t to say anything is wrong with the original version but my coding “abilities” has changed so I wanted to make it more streamline and hopefully more modular.

    In the original sniffMyPackets I used tshark to extract TCP and UDP streams, which wasn’t the ideal solution however it worked so.. This time around I’m hoping to cut out using tshark completely so I’ve looking at what else Scapy can do. There is a “hidden” (well I didn’t know it was there) function called Sessions, this takes a pcap file and breaks it down into separate sessions (or streams). The plus side of this is that it’s actually really quick and doesn’t limit itself to just TCP & UDP based protocols.

    I’m still playing with the function at the moment but I’m hoping that I can use this for sniffMyPackets, so I thought I would share my findings with you.

    First off lets load a pcap file into Scapy.

    >>> a = rdpcap('blogexample.pcap')

    This is just a sample pcap file with a ICMP ping, DNS lookup and SSL connection to google.co.uk.

    If you want to check the pcap has loaded correctly you can now just run.

    >>> a.summary()

    Cool so lets load the packets into a new variable to hold each the sessions.

    >>> s = a.sessions()

    This loads the packets from the pcap file into the new variable as a dictionary object, which I confirmed by during.

    >>> print type(s)
    <type 'dict'>

    Now lets get a look at what we have got to work with now.

    >>> s
    {'UDP 192.168.11.228:21893 > 208.67.222.222:53': ...SNIP...

    That’s just the first session object from the pcap and it contains two parts (as required for a dictionary object). The first is a summary of the traffic, it contains the protocol, source IP address and port (split by a colon) and then the destination IP address and port (again split by a colon). The second part of the dictionary object is the actual packets that make up the session.

    Lets have a look at how to make use of the first part of the dictionary object (useful if you want a summary of the sessions a pcap file).

    >>> for k, v in s.iteritems():
    >>> print k
    >>>
    UDP 192.168.11.228:21893 > 208.67.222.222:53
    UDP 192.168.11.228:44710 > 208.67.222.222:53
    TCP 192.168.11.228:53482 > 208.180.20.97:443
    UDP 208.67.222.222:53 > 192.168.11.228:12321
    ...SNIP...

    This is a nice and easy iteration of the dictionary object to get all the key within it (the summary details). You can of course slice and dice that as you see fit to make it more presentable. The other part of the sessions dictionary is the actual packets, using a similar method we can iterate through the sessions dictionary and pull out all the raw packets.

    >>> for k, v in s.iteritems():
    >>> print v
    >>>

    This will output a whole dump of all the packets for all of the sessions which isn’t that great. Lets see if we can make it a bit more friendly on the eye.

    >>> for k, v in s.iteritems():
    >>> print k
    >>> print v.summary()
    >>>
    ...SNIP...
    TCP 208.180.20.97:443 > 192.168.11.228:53482
    Ether / IP / TCP 208.180.20.97:https > 192.168.11.228:53482 SA
    Ether / IP / TCP 208.180.20.97:https > 192.168.11.228:53482 A
    Ether / IP / TCP 208.180.20.97:https > 192.168.11.228:53482 A / Raw
    Ether / IP / TCP 208.180.20.97:https > 192.168.11.228:53482 A / Raw
    Ether / IP / TCP 208.180.20.97:https > 192.168.11.228:53482 PA / Raw
    Ether / IP / TCP 208.180.20.97:https > 192.168.11.228:53482 PA / Raw
    Ether / IP / TCP 208.180.20.97:https > 192.168.11.228:53482 PA / Raw
    Ether / IP / TCP 208.180.20.97:https > 192.168.11.228:53482 A
    Ether / IP / TCP 208.180.20.97:https > 192.168.11.228:53482 PA / Raw
    Ether / IP / TCP 208.180.20.97:https > 192.168.11.228:53482 A
    Ether / IP / TCP 208.180.20.97:https > 192.168.11.228:53482 PA / Raw
    Ether / IP / TCP 208.180.20.97:https > 192.168.11.228:53482 PA / Raw
    Ether / IP / TCP 208.180.20.97:https > 192.168.11.228:53482 A
    Ether / IP / TCP 208.180.20.97:https > 192.168.11.228:53482 PA / Raw
    Ether / IP / TCP 208.180.20.97:https > 192.168.11.228:53482 PA / Raw
    ...SNIP...

    Now isn’t that much better, as you can see from the snippet above the session it’s captured starts at the SYN/ACK flag (probably because I started my capture after the SYN was sent) so in reality (and with a bit more testing) it looks like the Scapy Sessions function does allow for extracting TCP streams (well all streams really).

    Lets have a go at something else.

    >>> for k, v in s.iteritems():
    >>> print v.time
    >>>
    AttributeError: 'list' object has no attribute 'time'
    >>>>

    Oh well I was hoping that I could get the time stamp for each packet but it seems that you need to do some extra work first.

    >>> for k, v in s.iteritems():
    >>> for p in v:
    >>> print p.time, k
    >>>
    1417505523.55 UDP 192.168.11.228:21893 > 208.67.222.222:53
    1417505522.45 UDP 192.168.11.228:44710 > 208.67.222.222:53

    Oh that works much better, because the packets in the second half of the dictionary are a list object we need to iterate over them to get access to any of the values within them (such as the packet time).

    So there you go, I’m off to see what else I can do with this..

    Enjoy.

    Posted in packets, Scapy

    Maltego: Going Remote..

    Today we are going to talk about TDS (Transform Distribution Server) transforms, as we discussed in earlier posts these bad boys run on a remote server and allow you to easily share your transforms with anyone (well people you want to).

    There are some considerations you need to bear in mind when building TDS transforms.

    The server you are hosting them on needs to have all the dependencies required to run (common sense really)
    If you are hosting transforms that are public and they spam the crap out of someone else’s systems you are probably the one going to get the blame
    Use a local development server to test them before pushing to live and think about using a private code repository (trust me it’s painful otherwise)

    You also need to think about what you want your TDS transform to do, for example TDS transforms (Public ones) are great for web-based queries (API calls etc) but you probably wouldn’t want to run anything that was CPU intensive or needed an asset uploaded from your local machine in order to work. You can purchase an internal TDS server from Paterva which might remove some of these restrictions (not sure how they work to be honest).

    I’ve not written a lot of TDS transforms (at the moment) but in this post we are going to take the GetRobots local transform we’ve been working on and move it to a TDS server. In order to use TDS transforms you need the following:

    I’m not going to cover building a web server to host the WSGI application as the documentation linked above does a pretty good job (it worked for me so you should be fine). What we are going to look at today is how to take the local transform and convert it to a TDS transform and what that means in terms of code changes (it’s not that much actually).

    So going back to the very first iteration of the GetRobots transform, lets just have a look at the code we wrote.

    #!/usr/bin/env python
    
    # Maltego transform for getting the robots.txt file from websites
    
    from MaltegoTransform import *
    import os
    import requests
    
    website = sys.argv[1]
    m = MaltegoTransform()
    robots = []
    
    try:
      r = requests.get('http://' + website + '/robots.txt')
      if r.status_code == 200:
        robots = str(r.text).split('\n')
        for i in robots:
          m.addEntity('maltego.Phrase', i)
      else:
        m.addUIMessage("No Robots.txt found..")
    except Exception as e:
      m.addUIMessage(str(e))
    
    m.returnOutput()
    

    Only 24 lines of code, the good news is that we are not going to have to change a lot to make this a TDS transform. Lets jump straight in and see what the TDS version will look like.

    import requests
    from Maltego import *
    
    robots = []
    
    def trx_GetMeRobots(m):
      TRX = MaltegoTransform()
      try:
        website = m.Value
        TRX.addUIMessage(str(website))
        url = 'http://' + website + '/robots.txt'
        TRX.addUIMessage(str(url))
        r = requests.get(url)
        if r.status_code == 200:
          robots = str(r.text).split('\n')
          for i in robots:
            ent = TRX.addEntity('maltego.Phrase', i)
          else:
            TRX.addUIMessage("No Robots.txt found..")
      except Exception as msg:
        TRX.addUIMessage("Error:"+str(msg),UIM_PARTIAL)
             
      return TRX.returnOutput()
    

    See told you it wasn’t that bad, lets look at the important changes.

    from Maltego import *

    Here we have change the Maltego module we import, the TDS based transforms use a different Python library (which has better functionality in some areas).

    def trx_GetMeRobots(m):

    For the TDS transforms we use a Python function, the name is of your choice but you need to ensure you have the ‘(m)’ defined as this is used to pass the entity value(s) from your Maltego graph into the transform.

    TRX = MaltegoTransform()

    We’ve used TRX as the variable for the MaltegoTransform() call rather than ‘m’ as we already used ‘m’ in the function and it helps me remember its a TDS transform. Again we use a try/except loop and as you can see the only real difference is that I’ve added some ‘UIMessage’ updates along the way (which was more for debugging).

    TRX.addUIMessage(str(website))
    TRX.addUIMessage(str(url))

    So hopefully by this point you should have the following:

    • A web server configured to work with the TDS package
    • Your new TDS transform written

    The next part is to tweak the TRX.wsgi code to make use of our new transform. The code is below:

    import os,sys
    
    # To run in debug, comment the next two lines and see 
    # end of this file
    
    os.chdir(os.path.dirname(__file__))
    sys.path.append(os.path.dirname(__file__))
    
    from bottle import *
    from Maltego import *
    
    # Transform Libraries goes here:
    from GetRobots import *
    
    
    ##### TRANSFORM DISPATCHER starts here -->
    
    #-----> GetRobots - Collect contents of robots.txt from a website
    @route('/GetRobots', method='GET')
    def GetMeRobots():
      if request.body.len>0:
        return(trx_GetMeRobots(MaltegoMsg(request.body.getvalue())))
    
    
    ## ---> Start Server
    ## To start in debug mode: Comment the line below...
    application = default_app()
    
    ## ... and uncomment line below
    #run(host='0.0.0.0', port=9001, debug=True, reloader=True)
    

    Most of the code above is the default so I’m just going to cover the bits I’ve changed. The first bit is this:

    from GetRobots import *

    Essentially this imports our GetRobots code so it’s available to be called by the transform dispatcher. Next we create a new route so we can call the transform from within Maltego.

    #-----&gt; GetRobots - Collect contents of robots.txt from a website
    @route('/GetRobots', method='GET')
    def GetMeRobots():
    if request.body.len&gt;0:
    return(trx_GetMeRobots(MaltegoMsg(request.body.getvalue())))

    The ‘@route(‘/GetRobots’, method=’GET’)’ statement is the URL that you want to reference your transform on, the default for the ‘method’ is “ANY” but I’ve changed it to “GET” so it’s not as open. The ‘def GetMeRobots():’ function is just to hold our call to the transform, the first line of the function ‘if request.body.len>0:’ makes sure that we only perform the call to the transform if the body length isn’t blank (or zero).

    The important part of the next line of code is the call to your transform (based on function name) ‘return(trx_GetMeRobots(MaltegoMsg(request.body.getvalue())))’. You need to ensure that the name of the function, in this case ‘trx_GetMeRobots’ matches then name of your function for your transform.

    Still with me? So we now have the following all setup and ready to roll.

    • A web server configured to work with the TDS package
    • Your new TDS transform written
    • A modified TRX.wsgi configured for your new transform
    • So there are two steps left, we now need (using our TDS account) to set up the Seed & Transform. This is a quick process. Log into your TDS account (https://cetas.paterva.com/TDS/)

      Screen Shot 2014-10-08 at 12.23.51

      Select “Seeds” from the menu, then click on the “Add Seed” button.

      Screen Shot 2014-10-08 at 12.24.51

      We need to fill in some basic details.

      Seed Name – A name for your seed, you can create seeds based on the transform roles or environments
      Seed URL – Create a unique name which will act as the path to your transforms. If you want them “private” make the URL suitably long and stupid.
      Transforms – This will be blank at the moment

      Click on “Add Seed” and then all being well you should get a message saying the seed was added successfully. Next we need to add the transform, there isn’t an option for “Home” so you need to click on the hyperlink at the top of the page labelled ‘Paterva Transform Distribution Server 0.1′.

      Screen Shot 2014-10-08 at 12.48.46

      Back on the home page, click on ‘Transforms’, then click on “Add Transform”.

      Screen Shot 2014-10-08 at 12.50.41

      We now need to add in some details, we will do the minimum just to save on time (and me typing).

      Transform Name – The name of the transform. It doesn’t support spaces or non alpha numeric characters.
      Transform URL – This is the URL to YOUR server running the transform and the name of the path to the transform (based on what you entered in the TRX.wsgi file).
      Input Entity – For our purpose that is maltego.Website, you can add your own entities as well
      Owner – This is based on your TDS account details.
      Disclaimer – By default a disclaimer is added but you can add your own text as well.
      Description – An optional area to put a more detail about what your transform does.
      Version – Again optional, good if you are making big changes just so people are aware.
      Author – Again pulls these details from your TDS account.
      Debug – Tick to enable debugging (explains itself really).
      Transform Settings – We haven’t defined any and I haven’t worked out what they are really for.
      Seeds – You need to select the seed you created, this adds the transform to that seed.

      Screen Shot 2014-10-08 at 13.03.14

      Click “Add Transform”, and again you should get a message something like ‘Successfully Added [TransformName]’

      The final task is to add the Seed into Maltego so it will add the transform and make it available. This is a nice simple process and you just need the full Seed URL, here’s one I built earlier.

      https://cetas.paterva.com/TDS/runner/showseed/49f742b02c27ab2a88b4299a4fb56f31abc6298c

      Load up Maltego and click:

      Manage – Discover Transforms – Discover Transforms (advanced)

      Screen Shot 2014-10-08 at 13.06.23

      You will now get a little wizard which only needs two pieces of information.

      Name – You can call this what you what
      URL – The Seed URL

      Screen Shot 2014-10-08 at 13.08.24

      Click Add and then Next

      Screen Shot 2014-10-08 at 13.08.31

      The wizard will go off and verify all the Seeds are available and then present you with a list.

      Screen Shot 2014-10-08 at 13.08.57

      All you need to do on this is click Next to carry on (they should all be ticked). This will produce a list of all the transforms that will be loaded into Maltego. Scroll through the list and you should be able to see this.

      Screen Shot 2014-10-08 at 13.11.11

      Click Next and it will import tha transforms. All being well you should get a similar window to the one below and that is it.

      Screen Shot 2014-10-08 at 13.12.22

      Now to test it, create a new graph and drag a Website entity onto it, right-click and select.

      Run Transform – Other transforms – GetRobots

      Screen Shot 2014-10-08 at 13.13.25

      The first time you run a TDS transform you will be prompted to accept the disclaimer which if you want it to run you should accept.

      Screen Shot 2014-10-08 at 13.14.41

      This will run the GetRobots transform from the remote TDS server and should produce the same results as the local transform we wrote.

      Screen Shot 2014-10-08 at 13.15.41

      So there you go, your first TDS transform now wasn’t that fun. The final blog post in the Maltego series will go over exporting your entities, transform sets and the such so that other people can use them.

    Posted in Maltego | Tagged

    Maltego: Creating entities..

    So far in this blog series we have talked about how you can create new Maltego transforms and machines easily. Today we are going to cover creating new entities for you to use and/or abuse.

    Creating new entities is (like creating transforms) quite easy and painless, there are a couple of things you need to keep in mind but we will cover those as we go.

    To create a new entity within Maltego click:

    Manage – Manage Entities

    Screen Shot 2014-09-29 at 12.55.55

    You will now get the Entity Manager window.

    Screen Shot 2014-09-29 at 12.56.36

    Once again (like most things in Maltego) you get a ‘New Entity Wizard‘. Lets have a look at the options you have to supply.

    • Display name – This is the name of your entity as it will appear in the ‘Palette‘ within Maltego, try to keep it short and sweet
    • Short description – A quick one liner of what the entity is for
    • Unique type name – Again like your transform make it unique and relevant to the entity type
    • Inheritance
      • Base Entity – Inheritance allows you to create a new entity but ‘inherit‘ the properties of an existing entity. This means if you want to create a new entity for ‘IPv4 Address‘ which has additional fields for your needs you can ‘inherit‘ the base properties and the transforms that can be run against the base entity into your new entity (it’s actually a really cool feature).
    • Icons – For each new entity you create you have to set an icon. You can either use a built-in one or use a new one. For our purpose we are going to use an existing one.

    Click on Browse

    Screen Shot 2014-09-29 at 12.56.49

    Make sure you have selected the ‘Default’ tab at the top, you should now see a whole stack of icons you can pick.

    Screen Shot 2014-09-29 at 13.06.17

    Pick an icon, any icon and then click on ‘OK’. You should hop back to the Entity Wizard and your Icon image should now be populated with your choice.

    Screen Shot 2014-09-29 at 13.12.17

    Still with me?? Cool right-click ‘Next’ and we progress onto the next page. This is where we have to define the properties for the entity. If you use Inheritance this is typically grayed out as it will want to use the existing properties from the entity you are inheriting from (but you can add additional ones if you like). Here we need to define the main property for the entity, this is the one that we will set by default each time we return this entity type.

    Screen Shot 2014-09-29 at 13.14.37

    A lot of the fields are set by default but we can go through them just so you know what they are for.

    • Property display name – This is the name of the property when you open the entity. I tend to keep it the same as the Display name as I set as this is the main property and it just makes sense to me to have them the same.
    • Short description – This is a short description of the entity property (rather than the description of the entity)
    • Unique property name – I have a habit of setting this to the same as the entity unique type name (when using Canari) but this is what you will need to call in your transform code when setting the value.
    • Data type – By default this is a ‘string’ value, however you can set the following types:
      • string
      • int
      • date
      • double
    • Sample value – If you drag your entity into a graph, this is the value that will get set and display for this property.

    We will just tweak a couple of values before moving on.

    Screen Shot 2014-09-29 at 13.24.01

    Now we are done, click on ‘Next’ and lets carry on. We can now add our new entity into an existing or new category.

    Screen Shot 2014-09-29 at 13.24.27

    You can add your new entity into one of the existing categories which are listed below (and are the ones listed in your palette within Maltego).

    • Devices
    • Infrastructure
    • Locations
    • Penetration Testing
    • Personal
    • Social Network

    For our purpose we are going to create a new category called ‘MyFirstTransform‘. If you want to create a new category just type it into the box.

    Screen Shot 2014-09-29 at 13.26.50

    Click Finish.

    All being well in your ‘Palette‘ pane on the right you should see the new category called ‘MyFirstTransform‘ and in the entity list you should see the ‘Robot Entry‘ entity.

    Screen Shot 2014-09-29 at 13.27.41

    Before we move on I just want to cover creating additional properties for our new entity. This is handy if you want to use more than one property in executing a transform or storing in your entity. From within the Entity Manager click on the 3 little blue dots on the right of the new entity.

    Screen Shot 2014-09-29 at 13.30.57

    Click on the ‘Additional Properties‘ tab at the top, you will see the original property that we created so we are going to add another (just for giggles). Click on ‘Add Property‘ in the top right.

    Screen Shot 2014-09-29 at 14.20.09

    We are going to add a new property for the dynamic property we called in the origin GetRobots-mk2 transform. Type the following into the available fields.

    Name – properties.orgurl
    Display Name – Original URL
    Type – String

    Screen Shot 2014-09-29 at 14.21.59

    The field name is used for the unique name, and display name is the “friendly description”.

    You may notice that the list of available types is a lot longer then when we created the entity. I’m not sure why but it is, ‘string’ for example has to options:

    • string
    • string[]

    The ‘string’ type is for a single string value, while the ‘string[]’ allows you to provide a range of string values (like the ‘ports’ property in the Website entity).

    Click OK to add the new property and then click OK again to return to the Entity Manager.

    Screen Shot 2014-09-29 at 14.26.11

    Awesome, isn’t this fun. Now we need to tweak our existing transform to make use of the new entity. Lets have a look at the code again to refresh our memory.

    #!/usr/bin/env python
    
    # Maltego transform for getting the robots.txt file from websites
    
    from MaltegoTransform import *
    import requests
    
    m = MaltegoTransform()
    m.parseArguments(sys.argv)
    
    website = m.getVar('fqdn')
    port = m.getVar('ports')
    port = port.split(',')
    ssl = m.getVar('website.ssl-enabled')
    robots = []
    
    try:
      for c in port:
        if ssl == 'true':
          url = 'https://' + website + ':' + str(c) + '/robots.txt'
          r = requests.get(url)
          if r.status_code == 200:
            robots = str(r.text).split('\n')
            for i in robots:
              ent = m.addEntity('maltego.Phrase', i)
              ent.addAdditionalFields("url","Original URL",True,url)
          else:
            m.addUIMessage("No Robots.txt found..")
        else:
          url = 'http://' + website + ':' + str(c) + '/robots.txt'
          r = requests.get(url)
          if r.status_code == 200:
            robots = str(r.text).split('\n')
            for i in robots:
              ent = m.addEntity('maltego.Phrase', i)
              ent.addAdditionalFields("url","Original URL",True,url)
          else:
            m.addUIMessage("No Robots.txt found..")
    except Exception as e:
      m.addUIMessage(str(e))
    
    m.returnOutput()

    To make use of the new entity we just need to change two lines of code (there are two instances of this though). The original lines code of code that define your entity returned is below:

    ent = m.addEntity('maltego.Phrase', i)
    ent.addAdditionalFields("url","Original URL",True,url)

    We simply need to change this to our new entity type and property values.

    ent = m.addEntity('adammax.robotentry', i)
    ent.addAdditionalFields("property.orgurl","Original URL",True,url)

    This is what the code should look like now.

    #!/usr/bin/env python
    
    # Maltego transform for getting the robots.txt file from websites
    
    from MaltegoTransform import *
    import requests
    
    m = MaltegoTransform()
    m.parseArguments(sys.argv)
    
    website = m.getVar('fqdn')
    port = m.getVar('ports')
    port = port.split(',')
    ssl = m.getVar('website.ssl-enabled')
    robots = []
    
    try:
      for c in port:
        if ssl == 'true':
          url = 'https://' + website + ':' + str(c) + '/robots.txt'
          r = requests.get(url)
          if r.status_code == 200:
            robots = str(r.text).split('\n')
            for i in robots:
              ent = m.addEntity('adammax.robotentry', i)
              ent.addAdditionalFields("property.orgurl","Original URL",True,url)
          else:
            m.addUIMessage("No Robots.txt found..")
        else:
          url = 'http://' + website + ':' + str(c) + '/robots.txt'
          r = requests.get(url)
          if r.status_code == 200:
            robots = str(r.text).split('\n')
            for i in robots:
              ent = m.addEntity('adammax.robotentry', i)
              ent.addAdditionalFields("property.orgurl","Original URL",True,url)
          else:
            m.addUIMessage("No Robots.txt found..")
    except Exception as e:
      m.addUIMessage(str(e))
    
    m.returnOutput()

    Save the changes and then within Maltego run the GetRobots transform again, and you should see your new entity spring into life (fingers crossed).

    Screen Shot 2014-09-29 at 14.35.50

    So there you go, the basics behind creating your own entities. Next time we are going to talk about exporting all the Maltego goodness we have created.

    Posted in Maltego | Tagged

    Maltego: Rise of the Machines…

    In this episode of Maltego loving (or learning if you prefer that) we are going to talk about Maltego Machines. Maltego machines in the simplest form are a collection of transforms that execute in order, each one feeding off the entities generated from the last transform.

    Maltego Machines to start with may seem complicated especially when you are trying to get them to run in parallel, but they are incredibly useful and powerful so it seemed worthy to give them their own blog post.

    The official documentation for Maltego Machines can be found HERE.

    To start with we are going to create a simple Maltego Machine to run the ‘GetRobots-mk2′ transform we created in the earlier blog posts. In reality a Machine to run one transform is a bit of a waste but it’s a good place to start. To find the Machines within Maltego click.

    Machines – New Machine

    This will launch a wizard that we can use to create our first Machine.

    Screen Shot 2014-09-25 at 07.18.06

    Lets fill in the details;

    Display Name – The display name for our Machine.
    Unique ID – This is the same principle as with Transforms, make it unique and relevant.
    Author – That will be you.
    Description – Write something in that will make sense if anyone else uses your Machine.

    Screen Shot 2014-09-25 at 07.20.39

    Click Next….

    Now we get a choice of the type of Machine we want to create. There are two main types (excluding blank, which we will come back to).

    Macro – These Machines will execute once (and only once) when you start them.
    Timer – These are my favourite Machines, once started they execute in a timed loop (defined by you) and will keep running until you stop them.

    Screen Shot 2014-09-25 at 07.21.01

    Select Macro and click Next…

    Screen Shot 2014-09-25 at 07.24.38

    Now we get to the good part, the next window is the main Machine ‘editor’, this is where you can add your transforms and make sure the Machine compiles properly. You will also notice that they have pre-loaded the Machine with some example transforms so you have an idea of what it should look like.

    There are three options along the top which we will talk about first.

    Format – This essentially formats your code so it looks pretty.
    Toggle Comment – Will either comment a line or un-comment it (saves you typing 2 //’s).
    Compile – Checks your code and makes sure that it is all happy.

    Screen Shot 2014-09-25 at 07.26.13

    On the right hand side of the window is the list of transforms available for your Machine, you can add local transforms or remote transforms (TDS), but bear in mind if you want to share your Machines you need to make sure that everyone has the transforms you used available (more on exporting stuff from Maltego in the next post).

    Screen Shot 2014-09-25 at 07.48.14

    Right let’s get on with the actual coding of the Machine shall we, first off we are going to remove all the stuff that Maltego puts in there by default (that’s not actually needed).

    //Welcome to Maltego Machines!

    //Each machine starts with a statement like this
    machine("adammax.macgetrobots",
    displayName:"Get Robots.txt",
    author:"@catalyst256",
    description: "A simple 1 transform Maltego machine to uses \
    the GetRobots-mk2 transform.") {

    //A macro machine has a start function like this
    start {

    }
    }
    //Of course there is much more you can do with machines... Have fun!

    Right so we now have a basic (yet empty) Machine so lets add in our transform. In the space between ‘start {‘ and the following ‘}’ expand ‘Local’ on the right hand side and then double-click on GetRobots-mk2. Your code should now look like this.

    //Welcome to Maltego Machines!

    //Each machine starts with a statement like this
    machine("adammax.macgetrobots",
    displayName:"Get Robots.txt",
    author:"@catalyst256",
    description: "A simple 1 transform Maltego machine to uses \
    the GetRobots transform.") {

    //A macro machine has a start function like this
    start {
    run("adammax.GetRobotsMk2")

    }
    }
    //Of course there is much more you can do with machines... Have fun!

    Notice the lack of formatting?? Click on the Format button at the top of the window and magically your code will now look like this.

    //Welcome to Maltego Machines!

    //Each machine starts with a statement like this
    machine("adammax.macgetrobots",
    displayName:"Get Robots.txt",
    author:"@catalyst256",
    description: "A simple 1 transform Maltego machine to uses \
    the GetRobots transform.") {

    //A macro machine has a start function like this
    start {
    run("adammax.GetRobotsMk2")

    }
    }
    //Of course there is much more you can do with machines... Have fun!

    Ok so we’ve added the transform, we sorted the formatting so now we just need to check to make sure it will compile. Click the green tick at the top of the window and the compile debug screen will appear at the bottom of the window. If we haven’t made any horrible mistakes (just copy and paste the code if you have a problem), you should get this wonderful message.

    Compiling machine...
    Success

    OK so now we have the Machine written (yes thats it), click on Save….

    We can now run the Machine, create a new graph and add a Website onto it, right-click and under Machines you should see our newly created Machine. But wait what black magic is this? How does the Machine know it can run on that entity type?? So I have no fancy answer for this, other than it takes the first transform in the Machine and based on what entity it is designed to run on will then appear when you right-click on that entity type.

    Screen Shot 2014-09-25 at 08.15.08

    When a Machine is running you get a window in the top right of your Maltego screen that will show the status of the Machine and once it has completed will update and tell you how many entities returned.

    Screen Shot 2014-09-25 at 08.15.30

    Lets now have a look at some of the more “advanced” Machines that are available. We will look at some of the built-in Machines and work through how they work. Click on

    Machines – Manage Machines

    Screen Shot 2014-09-25 at 08.23.46

    As you can see there are already some pretty cool machines available by default. Lets open one up and see what makes it tick.

    Double click on the ‘Footprint L1′ Machine, and the Machine editor will open up.

    Screen Shot 2014-09-25 at 08.25.20

    Now these Machines are read-only so we can’t make changes to them, we can however (if so inclined) copy and paste them into a new Machine.

    Lets work through the code and see what this Machine does.

    machine("paterva.footprint.level1",
    displayName:"Footprint L1",
    author:"Roelof Temmingh",
    description: "This performs a level 1 (fast, basic) footprint of a domain.") {

    start {
    //do all the DNS enumeration
    log("Performing DNS enumeration",showEntities:false)
    status("Phase 1 - DNS enumeration")
    paths {
    run("paterva.v2.DomainToWebsite_DNS")
    run("paterva.v2.DomainToDNSName_DNSBrute",slider:500)
    run("paterva.v2.DomainToDNSName_ZT",slider:10000)
    run("paterva.v2.DomainToSOAInformation")
    run("paterva.v2.DomainToWebsiteDNS_SE",slider:255)
    }

    //here we end up with DNS names (and MX,NS,websites)
    //take it to IP address and Netblock
    log("Resolving to IP",showEntities:false)
    status("Phase 2 - Resolve DNS names")
    run("paterva.v2.DNSNameToIPAddress_DNS")

    //we now have IP adddresses
    status("Phase 3 - Netblocks and AS")
    log("Computing netblocks",showEntities:false)
    run("paterva.v2.IPAddressToNetblock_Cuts")
    log("Looking up AS",showEntities:false)
    run("paterva.v2.NetblockToAS_SS")

    }
    }

    Right you ready for this..

    So the first few lines of this Machine detail the information about the Machine, which is the same as we have in ours.

    machine("paterva.footprint.level1",
    displayName:"Footprint L1",
    author:"Roelof Temmingh",
    description: "This performs a level 1 (fast, basic) footprint of a domain.")

    We then move into the core of the Machine, the first function is a ‘log’. Log is used send a message to the GUI within the Machine window.

    log("Performing DNS enumeration",showEntities:false)

    With this particular ‘log’ function we post the message “Performing DNS enumeration” to the GUI, the ‘showEntities:false’ means that once run it won’t log the list of entities to the GUI. If you set this to ‘showEntities:true’ then it will list the entities that the transform will run against.

    Next we can see a ‘status’ function.

    status("Phase 1 - DNS enumeration")

    The ‘status’ function within a Maltego Machine sets the contents of the status function (“Phase 1 – DNS enumeration”) as the label on the Machine window. This is useful if you are running lots of different stages of transforms and want to be able to track the current running set of transforms.

    We can now move onto the main body of this Machine.

    paths {
    run("paterva.v2.DomainToWebsite_DNS")
    run("paterva.v2.DomainToDNSName_DNSBrute",slider:500)
    run("paterva.v2.DomainToDNSName_ZT",slider:10000)
    run("paterva.v2.DomainToSOAInformation")
    run("paterva.v2.DomainToWebsiteDNS_SE",slider:255)
    }

    The are two ways you can run transforms in a Machine, you can either run in parallel or in series. The use of ‘paths’ means that these transforms will run in parallel. Basically this means that each of the 5 will run at once against the base entity (so that’s 5 transforms run from within 1 Machine). and each transform will return their own entity type into the graph.

    The ‘slider’ function within the ‘run’ sets the maximum number of entities that can be returned by the transform. If you are using the community edition of Maltego like me (still looking for donations of a full license) that is 12. The transform below when run will only return 500 entities.

    run("paterva.v2.DomainToDNSName_DNSBrute",slider:500)

    We can now move onto the second block of code.

    //here we end up with DNS names (and MX,NS,websites)
    //take it to IP address and Netblock
    log("Resolving to IP",showEntities:false)
    status("Phase 2 - Resolve DNS names")
    run("paterva.v2.DNSNameToIPAddress_DNS")

    Again we use ‘log’ and ‘status’ to provide output to the Machine window, we then run another transform which will take the DNS entity returned from the first block of code and run this transform against them.

    run("paterva.v2.DNSNameToIPAddress_DNS")

    The last block of code is below.

    //we now have IP adddresses
    status("Phase 3 - Netblocks and AS")
    log("Computing netblocks",showEntities:false)
    run("paterva.v2.IPAddressToNetblock_Cuts")
    log("Looking up AS",showEntities:false)
    run("paterva.v2.NetblockToAS_SS")

    This takes the IP addresses from the last transform and then runs two more transforms (with ‘log’ and ‘status’ messages) in sequence.

    So now we know some more functions of Machines, lets update ours to do a little bit more. We are going to add some ‘log’ and ‘status’ messages. Have a look at the code below (which you can copy and paste over the original Machine we did).

    //Welcome to Maltego Machines!

    //Each machine starts with a statement like this
    machine("adammax.macgetrobots",
    displayName:"Get Robots.txt",
    author:"@catalyst256",
    description: "A simple 1 transform Maltego machine to uses \
    the GetRobots transform.") {

    //A macro machine has a start function like this
    start {
    log("Hunting for Robots.txt",showEntities:true)
    status("Connecting to Website")
    run("adammax.GetRobotsMk2")

    }
    }
    //Of course there is much more you can do with machines... Have fun!

    Below is a screenshot of what the Machine window now shows.

    Screen Shot 2014-09-25 at 12.06.30

    There is a lot more to Maltego Machines that I haven’t covered but hopefully you will have a good idea of what they can do and how to go about coding them. Have a look at the other built-in Machines and the documentation I’ve linked at the top of the post is very good.

    Enjoy

    Posted in Maltego | Tagged