Archive

Archive for the ‘Netscaler’ Category

Netscalers: Making sense of the cookie – the finale

So this is the final part to my Netscaler cookie series. If you haven’t read the other two blog posts you may want to just so this makes a bit of sense..

Part 1

Part 2

All make sense now?? (probably not but it’s polite to ask)..

Before I get started I just want to clear something up. I am in no way shape or form a programmer.. It’s one of those areas that up until recently has made my head hurt (and not just from banging my head on the desk a lot) but it is an area that I want to improve on and the best way for me to learn is to do.

So how do you end a series of blog posts about Netscaler cookies and how to decrypt them.. well you write a program to do it for you. I decided to use python to write my little decryption program as it will run on both Windows and Linux (I’ve even tested it to make sure) and it seems to be used a lot by InfoSec type people.

Now this is my first ever python program/script/application and in fact it’s the very first time I’ve ever written something like this (unless you count the macro I wrote in Word 7 that did a cypher substitution encryption), so yes while the code might not be perfect and possibly badly written the important thing is that it works. :)

Now before I get to the part where I give you the link to the script (is script the right word??) here’s how it works (in basic terms).

The script is designed to do 2 things, it accepts an Netscaler Cookie from the command line;

python nsccookiedecrypt.py NSC_rfse-gesfe-etsgsvs... (not the complete cookie)

It then runs two re.search functions to separate the cookie name (the Netscaler load balancer vserver name) and then the Server IP (IP address of the server your are persistent too).

Once it has these variables, it performs two decryption actions, the first is the cipher substitution to give you the real Server Name;

Service Name=qerd-fdred-dsrfrur-erdded

It then runs the XOR decryption based on the key that was mentioned in Part 2 of my series to give you then Server IP;

Server IP=63.17.71.92

Currently the script outputs both to the command line, it’s not exactly high end coding but it’s not a bad start for me.

You can find the script HERE, I’ve tested in on over a dozen real life Netscaler Cookies, so I’m 90% happy it will work in all cases, it doesn’t use any fancy imports so you should be good to go with just a standard python install.

If you find any bugs or want to let me know how to make it better, please drop me a line. Over time once I get better at coding I will probably improve it. I’ve created a new “Page” on my blog with links to the code and hopefully over time I will add to it.

If you want to modify the script for your own uses, please do, however if you let me know so I can keep tabs on how it’s being used and what I can do to improve it.

I would like to thank Alejandro Nolla for inspiring me to write this (check out his load balancer finder) and Daniel Grootveld for helping me with the XOR decryption (and by help I mean stopping me from using a Excel spreadsheet).

Happy decrypting.

Netscalers: Making sense of the cookie – part 2

At the beginning of the week I wrote here about the Cookie’s that the Netscaler uses for persistence. In that post I explained how I discovered that the Cookie name was encrypted using a simple substitution cipher. The cookie value itself was encrypted to contain the Service IP (the IP of the server that your session sticks to) and the Service Port.

I assumed that this part of the cookie was encrypted using a “real” encryption method such as SHA-256 or some other similar cipher. I spent the next couple of days looking online to see if I could match the cookie length and output (it’s all Hex) to a cipher. In the end I gave up, not because it was too difficult but because I thought of a more cunning plan.. :)

This is an example Netscaler cookie (and by example I mean from a website on the internet);

NSC_wtsw-bmufsjbo-qvcmjd-iuuq=ffffffffaf18363b45525d5f4f58455e445a4a423660

My previous post dealt with how the “encrypted” cookie name was formed (that’s the bit up to the ‘=’), this post is about the 8 characters after the ffffffff (everything else after that apart from the last 4 characters seems to be padding).

This is what I knew about the encrypted values:

1. The cookie started with ffffffff which I believed was not required to identify the Service IP.
2. The output was Hex, so I assumed that there must be some way to reverse engineer the encryption back to the real IP.
3. The encrypted value for each octet of the IP address was not encrypted using the same method (I knew that because when looking at cookie value I could see the same IP octet encrypted to different values in the cookie).
4. The encrypted values were consistent across different Netscalers (ruling out the encryption being based on appliance specific details i.e. hostname or MAC address).

In order to decrypt the Service IP out of the cookie I could decided that using a VPX (Virtual Netscaler) I could generate a cookie value for each of the 255 IP address in each octet, armed with the power of Excel and Notepad I generated the necessary Netscaler config to create my samples and then using this command on the Netscaler;

show lb vserver [vserver name] | more

This allowed me to see each server and the matching Netscaler cookie value. I started entering these into Excel with the “real” IP value. I had worked through about 60 of the last octet (starting at x.x.x.0) when I realised that I was seeing a pattern. To work out the pattern I took a wild guess (they are the best sometimes) and tried this in Excel;

=HEX2DEC(CELL)-Real Value

This was the breakthrough I was looking for.. and here’s why

On the last octet of the IP address the Hex value 11 was really 0 if you the formula above you get the result “17”, use this formula for the next 16 real values (remember I have collected 60 already from earlier) and you see the following pattern:

Real Value    Difference
0                17
1                15
2                17
3                15
4                17
5                15
6                17
7                15
8                17
9                15
10               17
11               15
12               17
13               15
14               17
15               15

Carry on for another 16 and you find this:

Real Value      Difference
16                -15
17                -17
18                -15
19                -17
20                -15
21                -17
22                -15
23                -17
24                -15
25                -17
26                -15
27                -17
28                -15
29                -17
30                -15
31                -17

The next 16 after this repeated first example, in fact all of the decryption for each octet required a repeating pattern, I just needed to find the key. Before rushing ahead I used the 2 patterns above to fill the remaining last octet of 255 addresses but I swapped the formula to create the Netscaler Hex value (and save myself sometime);

=DEC2HEX(Difference+Real Value)

I then double checked this was correct by looking at my other generated cookie values and checking some from another 2 Netscalers that use this method in “live”. I was one happy geek, I then needed to do the same pattern matching for the other 3 octets, but because I knew I was looking for a pattern I only needed to generate a smaller sample set to work with.

Whereas the first pattern I discovered was based on chunks of 16 the others weren’t, the first octet is using the numbers 1 & 3 in chunks of 4 (and the negative values for these as well), the second octet is just based on 8’s in chunks of 8(+8 and -8), and the third was totally random (not the pattern, more the logic behind it) and work on 2,6,10,14,18,22,26 & 30 in chunks of 16 again(and then the negative versions).

Rather than boring you with pages of information I’ve produced a PDF with it all in here.

So I’ve tested this as much as I can, and it works, the cookies I’ve looked at (where I know the Service IP) matches against this decryption sheet and again that is over 4 different Netscalers, running different appliances, IP addresses and versions of firmware.

Once I learn how to write in some sort of programming language I am hoping to write this into an application, where you can input the cookie value and it will provide you the decrypted values, I can think of a couple of uses outside of Netscaler administration and I’m sure any Pen Testers/Ethical Hackers reading this can probably think of a few more.. :)

So to recap, I now know how to decrypt the Load Balancer name from the Cookie name and the Server IP from the Cookie value, the remaining part is the Service Port but I’m not too worried about that (at the moment) as I know that it if a Netscaler cookie ends 3660 then it’s port 80.

Let me know if you have any questions or feel that my maths is wrong somewhere along the line..

Happy cookie decrypting.

The Geek

Netscaler: Making sense of the Cookie – part 1

Today was the first day back after my Christmas break, so it was a bit “slow”. Never to sit around being bored, I was writing up some notes on Netscaler cookie’s for an ethical hacker called Alejandro Nolla who has written up a cool application for checking to see if a domain has a load balancer behind it. You can find the application here or follow Alejandro on Twitter at @z0mbiehunt3r

Anyway while typing up my info I discovered something about the Netscaler cookies that I hadn’t noticed before. The Netscaler cookies are by default “encrypted” in 3 parts. Below is the extract from Citrix regarding Netscaler cookies:

The format of the cookie that the NetScaler appliance inserts is:
NSC_XXXX=
where:
NSC_XXXX is the virtual server ID that is derived from the virtual server name.
ServiceIP is an encrypted representation of the service IP address.
ServicePort is an encrypted representation of the service port.

So the 3 “encrypted” parts are Virtual Server ID, ServiceIP and then ServicePort. After a bit of coffee I realised something about the Virtual Server ID, it is “encrypted” using a substitution cipher, for example a=z, d=c etc. etc. the name “NSC_mc_udru” would be “NSC_lb_test” on the Netscaler as the configured load balancer name.

Now it might not seem much to you, but I was happy with my discovery, my next challenge finding out how the ServiceIP and ServicePort is encrypted. This is an NSC cookie

ffffffff3c19594d45525d5f4f58455e445a4a423660

Now to me at first it looked like HEX the first 8 F’s equally to 255 255 255 255 which seemed like it was a subnet address you use to reference a single host (as you would expect from a persistence cookie), I also know that af when converted from hex to dec equals 175 but the server IP actually starts with 172. I’ve converted the rest from hex to dec but the numbers are out for the server IP. At the end of the example above I know that for port 80 (http) the value is 3660 only changes if the port changes, the rest seems to stay the same.

So I’m a third of the way there.. maybe I will never break the encryption but it’s fun trying and it’s given my brain a good workout. If you can spot something I’ve missed then let me know.

NOTE: I have since worked out how to decrypt the whole cookie and wrapped it up in a python script which you can find HERE

The Geek

Categories: Cryptography, Netscaler

Netscaler: rpcNode issue

A few weeks ago I upgraded the firmware on our front facing Netscaler appliances. I have a love/hate relationship with Citrix firmware upgrades, the last one managed to break the hardware on one of the appliances which resulted in it re-ordering the network interfaces (a bit annoying when it’s 2am). This time round the firmware applied fine however on my post implementation review I discovered the following error in the Netscaler logs (ns.log and auth.log which can be found in /var/log).

Oct 31 09:52:00 hostname sshd[8640]: Failed password for #nsinternal# from nsip port 16051 ssh2
Oct 31 09:52:00 hostname sshd[8641]: Failed password for #nsinternal# from nsip port 8474 ssh2

The impact of this error was that SSL certificates couldn’t be applied to the appliance as it responded with an error, other than that the appliance was working fine and HA failover was still functioning as expected (however the error moved when failed over to the secondary).

I logged a call with Citrix Support who on checking the configuration that the rpcNode password was different;

1) IPAddress: NSIP Password: 87s9das8d7a8d0a9d8a9 SrcIP: NSIP Secure: ON
2) IPAddress: HA NSIP Password: 8c7d34kj3434m3kl34k3ll34k3 SrcIP: NSIP Secure: ON

Now neither myself or Citrix have any idea why this would have changed during a firmware upgrade but it had. Citrix’s recommendation was to break the HA pair and then recreate it. Awesome, however because these appliances are used heavily 24/7 it meant that I would have to use GSLB to fail our live websites over to the other datacentre and then mess around with HA at silly o’clock in the morning.

Not being one to always take the advice of others (but then who does) I decide to experiment with an idea, rather than breaking the HA I would simply just reset the rpcNode passwords on the primary appliance. My logic being that as the passwords were different already changing them wouldn’t matter.

Logging on via SSH (I love command line stuff), I executed the following commands on the primary Netscaler:

set rpcNode NSIP -secure yes -srcip NSIP -password cleartextpassword (sets the rpcNode password on the primary appliance)
set rpcNode HA Node -secure yes -srcip NSIP -password cleartextpassword (sets the rpcNode password for the HA node)

Once I had changed the rpcNode password I checked /var/log/auth.log and the previous failure message had gone and was replaced with:

Oct 31 12:14:00 hostname sshd[9968]: Accepted password for #nsinternal# from NSIP port 24449 ssh2
Oct 31 12:14:00 hostname sshd[9967]: Accepted password for #nsinternal# from NSIP port 16042 ssh2

The final test was adding a new SSL Certificate to the appliance which was succesful with no error.. SUCCESS

The Geek

Categories: Netscaler

Netscaler: Making your own dashboards

Welcome reader(s), as you will come to learn I will probably post a lot about Citrix Netscalers. The main reason for this is because where I work we have 9 Netscalers in total and I have the privilege of being the “Expert” on these wonderous hardware load balancers (ok that’s enough fluffy talk about Netscalers).

The Citrix Netscalers are a very good piece of kit in terms of what they do, one side that I personally think they are lacking is on the reporting. The appliances have some built-in reporting that allows you to see some historical information and a dashboard for “live” information.

Aside from the built-in reporting Citrix have released a product called Citrix Command Center, this allows you to centrally manage some functions of the Citrix Netscaler (or Citrix WANscaler) in one Dashboard. Command Center allows you so view service/service group/load balancer status, automatically download Netscaler config files from your appliances, record Appliance Events and Alerts as well as the ability to execute predefined or custom scripts from Command Center to your appliances.

Don’t get me wrong this is useful in it’s own right and is a good addition to your estate if you have Netscaler appliances, however I needed something a little less complex for our 24/7 Control Center to be able to see a read only portal for the relevant information needed for support.

The good thing about Command Center (in my opinion) is that it runs on Microsoft SQL Server, which means I can use my limited SQL skills to pull out the data I want for my dashboards and throw it into a web page (again I’m not a web developer). The main areas of focus for my dashboards where:

1. Last 50 Events
2. GSLB Service Status (we use GSLB for site fail-over)
3. Service/Service Group Status

NOTE: This article only covers the SQL part of making your own Dashboard, I will leave the web page design to my readers as I’ve only worked on IIS (unless there are a lot of requests for this).

My first task was to get my head around the database schema for Command Center, thanks to the wonder of Visio I “reverse engineered” the database into a Visio diagram so I could refer to it without having to go through each table. The database schema can be found HERE to save you the trouble (it’s a PNG to save any issues with not having Visio installed).

Right so we have our database, copy of the database schema and we know what we are looking for. Time to find that data and put it in the format I want (feel free to change this).

NOTE: I’ve started and ended each query with [Query Start] and [Query End] you do not need to include these in your query. Where I have entered something in UPPERCASE with a _ separating words needs to be replaced with your relevant information (just query the table to see all the results to get a better idea).

1. Last 20 Events – This is a very useful report, not only does it show the state changes but it also any config changes, logins etc etc. If someone reports a “Netscaler Issue” this is the first place I check and you can change to show as many events as you want. The SQL Query for this is a straight forward one as seen below.

[Query Start]
–Netscaler Command Center
–Last Top 50 Events displayed in date order (most recent first)
–Written by IT Geek 20/10/2011

select Top 50
–The +3600 on the DATEADD is to allow for Timezone change, it might not be necessary for your appliance
DATEADD(s, TTime/1000+3600, ‘19700101’) as [Date & Time],
–Within this CASE statement I change the native IP address of the Netscaler Appliance to a more “Friendly Name”
case Source
WHEN ‘NETSCALER_IP’ THEN ‘FRIENDLY_NAME’
–I’ve changed the headers to more friendly headings using the AS statement
End as [Source], text as [Events], entity as [Description]
–The table for the Events is called “Event”
from Event
–You can change this WHERE statement to exclude entries you don’t want to see or aren’t interested in or you can just remove it.
where entity not like ‘Power%’ and text not like ‘User: #nsinternal#’
order by [Date & Time] desc

[Query End]

Hopefully (and I have tested them) this should display in SQL Query as the last 50 events from Command Center.

2. GSLB Service Status – So in our configuration we have GSLB configured to allow our Active/Passive configuration to be failed over between our data centres. The GSLB dashboard shows which service on which site is either UP, Down, Out of Service or Going Out Of Service (these are the reported status for the Netscaler).

The SQL query for this is a bit more complex, on my dashboard I use one query for one site and another for the remote site and then just display them side by side.

[Query Start]
–Netscaler Command Center
–GSLB Service Status
–Written by IT Geek 20/10/2011

SELECT
*
FROM
(
SELECT
*,
ROW_NUMBER() OVER (PARTITION BY [Netscaler], [GSLB Name] ORDER BY [Last Polled Time] DESC) AS RecentFirst
FROM
(
Select DISTINCT
–Within this CASE statement I change the native IP address of the Netscaler Appliance to a more “Friendly Name”
–because they are a HA pair they are displayed as 2 IP’s for each site
case NSIP
WHEN ‘NETSCALER_IP’ THEN ‘FRIENDLY_NAME’
WHEN ‘NETSCALER_IP’ THEN ‘FRIENDLY_NAME’
ELSE ‘UNKNOWN’
END AS [Netscaler],
–The +3600 on the DATEADD is to allow for Timezone change, it might not be necessary for your appliance
DATEADD(s, EPTime/1000+3600, ‘19700101’) as [Last Polled Time],
SVCFULLNAME as [GSLB Name],
case svcstate
WHEN ‘4’ THEN ‘Out of Service’
WHEN ‘1’ THEN ‘Down’
WHEN ‘7’ THEN ‘Up’
WHEN ‘5’ THEN ‘Going Out of Service’
ELSE ‘UNKNOWN’
END AS Health,
SVCIP as [Internal IP],
SVCPORT as [Port]
from MESERVICES
) RAWDATA
) SEQUENCED
WHERE
SEQUENCED.RecentFirst = 1
AND
(
CASE
–Within this CASE I tell my SQL query to ignore GSLB services that below to the remote site and then
–vice versa (trust me it works)
WHEN [Netscaler] = ‘REMOTE_FRIENDLY_NAME’ AND [GSLB Name] NOT LIKE ‘GSLB_SERVICE_NAME_LOCAL_SITE’ THEN 1
WHEN [Netscaler] = ‘LOCAL_FRIENDLY_NAME’ AND [GSLB Name] NOT LIKE ‘GSLB_SERVICE_NAME_REMOTE_SITE’ THEN 1
ELSE 0 END
) = 1
AND
–This AND statement is used to only show the site you are interested in, I use a ‘%name%’ query to specify
–but that depends on your naming convention
[GSLB Name] like ‘DEFINE_WHICH_SITE YOU CAN TO CHECK AGAINST’
order by [GSLB Name], [Netscaler]

[Query End]

I admit this query might not make sense when you look at it here, but if you want to use it then drop me an email and I would be happy to help sort out my ramblings into something sensible.

3. Service/Service Group Status – This last query allows me to check service group members status, as well as showing me the relevant server IP and port details. It comes in handy when we get complaints about something not working.

[Query Start]
–Netscaler Command Center
–Service Group member status
–Written by IT Geek 20/10/2011
SELECT
*
FROM
(
SELECT
*,
ROW_NUMBER() OVER (PARTITION BY [Netscaler], [Service Group Name], [Server IP] ORDER BY [Last Polled Time] DESC) AS RecentFirst
FROM
(
Select DISTINCT
–Within this CASE statement I change the native IP address of the Netscaler Appliance to a more “Friendly Name”
–because they are a HA pair they are displayed as 2 IP’s for each site
case NSIP
WHEN ‘NETSCALER_IP’ THEN ‘FRIENDLY_NAME’
WHEN ‘NETSCALER_IP’ THEN ‘FRIENDLY_NAME’
WHEN ‘NETSCALER_IP’ THEN ‘FRIENDLY_NAME’
WHEN ‘NETSCALER_IP’ THEN ‘FRIENDLY_NAME’
ELSE ‘UNKNOWN’
END AS [Netscaler],
–The +3600 on the DATEADD is to allow for Timezone change, it might not be necessary for your appliance
DATEADD(s, EPTime/1000+3600, ‘19700101’) as [Last Polled Time],
SVCGRPFULLNAME as [Service Group Name], SVCGRPMMBRIP as [Server IP], SVCGRPMMBRPORT as [Server Port],
case SVCGRPMMBRSTATE
WHEN ‘4’ THEN ‘Out of Service’
WHEN ‘1’ THEN ‘Down’
WHEN ‘7’ THEN ‘Up’
WHEN ‘5’ THEN ‘Going Out of Service’
ELSE ‘UNKNOWN’
END AS Health
from MESVCGROUP
) RAWDATA
) SEQUENCED
WHERE
SEQUENCED.RecentFirst = 1
AND
[Netscaler] not like ‘UNKNOWN’
order by [Service Group Name], [Netscaler], [Last Polled Time] desc

[Query End]

You will have to excuse me if my commenting isn’t up to standard but as I’m not use to doing it I wasn’t sure what to include. Any questions please let me know and I will be happy to help.

Hope this can be of help to you (only if you have Netscalers).

The Geek

Categories: General, Netscaler

Netscaler: Basic Command Cheat Sheet

Over the last few years I’ve spent a lot of time working with Citrix Netscalers, check out www.citrix.com for more details (yes I know I’m lazy). A lot of this work I do via the command line, yes there is a nice GUI available but when I sat the Netscaler course the instructor told us that the command line was quicker so that’s what I use a lot of the time. It also allows for commands to be scripted and/or copied and pasted if you are making bulk changes.

Next month my current employer has its annual DR test, because the Netscalers here manage 99% of the e-commerce traffic (and we are e-commerce mad), there is a requirement to take along a Netscaler appliance. The problem with this is 2 fold, 1) I’m on holiday that week (wasn’t planned, promise) and 2) we don’t have any spare physical Netscalers to take.

The solution to these problems is to take a virtual Netscaler (or a VPX appliance) which will run on VMware. With a bit of IT Geek black magic I have ported the “live” configuration over to a VPX. The other part of the problem (i.e. me not being around) is for one of the other IT guys to take a stack of notes (once I’ve written them) on what needs tweaking to get it to work. This prompted me to develop a basic command cheat sheet to save me a bit of work so I thought I would share it HERE.

Bear in mind it’s just some of the more basic commands and the ones that I feel will be relevant to the DR test. Who knows you might find a use for it, if you have Netscalers where you work. You can download a whole host of PDF documents from Citrix including a document listing all the command line options, but that’s a lot of reading.. :)

Enjoy

The Geek

Categories: Cheat Sheets, Netscaler
Follow

Get every new post delivered to your Inbox.

Join 825 other followers