Cloudera CDH4 Install on Cluster

So I talked about getting a cluster ready for Cloudera CDH4 free install yesterday, now I am going to talk about actually installing hadoop on the cluster.  You can see the setup article here.  Again, this is basically my install notes, if I do it again, I shall take screen shots to share.

1. Set up Yum to download (optional)
vi /etc/yum.conf
--add the following line
export http_proxy --created as variable within linux

2. Download Cloudera CDH4

3. Install CDH4
chmod u+x cloudera-manager-installer.bin
Wait for the install
4. Log into the web portal at the link provided by the installer
username admin
password admin

5. Choose the Cloudera Standard Install (Free)
Click Continue on the installer detail screen.
Specify hosts for your CDH cluster installation, list each node on a new line with fully qualified name.
click Search, if any errors, go fix your host files.
click Continue if no errors.

6. Choose packages
Use packages if just downloading initially
Use parcels if you want to make your own store to have a named/saved version for adding single nodes later without upgrading.  With this option you need to pick a version or download a copy and host yourself to upgrade from. 
Choose the Versions you want or don't.
Click Continue

7. Cluster Installation
choose root or user and enter your password.
Click Continue

8. Watch Install
Watch the spinning circles and pray nothing goes poorly.
Hit Continue.

9. Look at errors you have to fix

10. Inspect Role assignments
Set your name node
set your secondary namenode
Set at least 3 zookeepers
Set all nodes that aren't name to tasktracker and data node.  
Gateway is the 2 name nodes
Job tracker is on secondar name node.
Hive meta store on namenode
Push HiveServer, Hue, Cloudera Manager, Service Monitor and all alerts on the name node.
All other services you probably wont use on secondary name node.
click Continue when you think it is configured correctly.

11. Database Setup
Choose your db types, default PostgreSQL.  
Save usernames passwords for later.
Test Connection

12. Review Server Configurations
Check Data Directories, this is the drives hdfs will use on the machines
Checking volumes on data nodes, namenodes, secondary name nodes
Ideally you don't have to change much here, just look for errors.  
Click continue,

13. Starting cluster services
It proceeds through setting up of the cluster, it takes a while and is the final step.

14. Now you should be at the Cloudera Manager Dashboard

Setting up a Hadoop Cluster for Hadoop Install

A project I have recently been a part of was setting up a POC hadoop cluster for our organization.  We did a simple 6 node cluster all using CentOS 6.4 and the free Cloudera CDH4.  I had installed this on a single node easy enough with the online tutorials, but felt a comprehensive list of tasks needed to prepare the cluster for install wasn't clearly laid out anywhere.  This is how we did it.  For now its just a list of instructions, I will come back an add screen shots if I ever do it again, this was mostly done through putty with vi for editing.

1. Disable the selinux firewall on all nodes.
ls etc/selinux --go to the folder
vi config --edit the config file
selinux=disable --to disable the linux firewall and allow ports to open up so nodes can talk to eachother.
sudo service network restart --to restart the adapter

2. Generate SSH Key - as root on name node
ssh-keygen -t rsa--enter a few times, key generated
cd .ssh -- to the ssh files generated
cp id_rsa authorized_keys --copy
ssh-copy-id hdfsnode2--copy the key to the node hdfsnode2 (computer network name)
ssh hdfsnode2should now allow login to that machine without password as the key is installed.
repeat on all machines in node

3. IP Table Services
service iptables status
--should see firewall is not running
chkconfig iptables off --now the nodes will not have this service start
repeat on each node
4. Make sure host files one each has all nodes

vi /etc/hosts --update host file to have reference to each node including itself ip fullname alias
scp /etc/hosts hdfsnode2:/etc/ --copy host file to each node

5. Restart all servers
ssh hdfsnode2 --get to other server
init 6 --does remote restart

Adding CKEditor To an MVC 3 Razor in C# with JQuery

I am creating a web application that will allow content manipulation by end users.  This requires me to insert some sort of editor for them to manage content.  I have used several of these in the past on a few of my ASP.Net sites, but this will the first time in MVC 4 with Razor and its been a few years so I decided to look around.  
I stumbled across CKEdit which is a JQuery based library for a simple HTML editor.  The code seemed laid out well so strapping it on MVC shouldn't be that hard, and it wasn't.  Here are the few steps it took me to get this working well.
1. Download CKEdit from their website.
From here you can even choose the layout you like.  It downloads as a zip file that extracts into a folder you should just place at the root level of your web application.
2. Add the new folder and content to you Project
In VS 2012, you must add each folder then the files in it, sort of annoying you can't add an existing folder, oh well.  Make sure to right click the folder and in properties, set it to always copy to output.  
3. Update your Layout file 
If you want this available on all your web pages do it here, otherwise these just go in the pages you need the editor.  Add or make sure these 3 lines are added in the header.  These include all the Javascript you need to make this happen.
4. Add this function JS Function on the page you want it rendered.
Again, this could be in an include file or in the layout page, or just on the page you need it.
<script type="text/javascript">
    $(function () {
5. Tie your model to the Editor.
So, the above JavaScript function will replace an object with the name content with the ckeditor.  The attribute on my model is named Content, so the id of it will be content, and so the above function finds it and places the editor over it.  I use the following code to generate a text area for it to replace, mostly because a text area is easy to fail to if javascript isn't working.
<div class="editor-label">
            @Html.LabelFor(model => model.Content)
        <div class="editor-field">
            @Html.TextAreaFor(model => model.Content)
            @Html.ValidationMessageFor(model => model.Content)
6. Update Controller
My controller now needs to be able to land my post with the model.  What is interesting about this is it will throw an error because of possible injection allowing all characters to be posted to the server.  To allow the editor to post the HTML, which it must or it will never render later, you need to put the [ValidateInput(false)] above the function that catches it.  After that, treat it like a string and save it away.
7. Render the saved content.
This should be as easy as just @Object.Field however it isn't of course.  MVC is smarter than us, and html encodes any string rendered so all tags will show up.  What we have to do instead is @Html.Raw(Object.Field) and everything will look great.  
8 Have Fun!

Shipping Girth Calculations in C#

I recently had to calculate girth in a shipping sense which isn't very exciting.  It requires you to take the length, width, and height of the object and use the following equation.  Longest Side + 2*(Middle Side + Shortest Side).  
I would like to do this in a line of code, but alas, determining which of the 3 variables is longest makes this a if else type problem.  Here is my solution, I thought it rather nice, though I'm sure more memory is used and it compiled larger than a simple if else setup, but I think it reads nicer.

var sl = new List<double>() { Length, Width, Height };
var Girth = sl.Max() + 2 * sl.OrderBy(c=>c).ToList()[1] + 2 * sl.Min();

The use of a List allows some macro functions like Max and Min, and I can use the .orderby function and some lambda to index through and get the middle sized item.

WCF ws-Security and SoapUI

So recently I talked about ws-Security Headers with WCF.  But today I ran into the problem of trying to test said service locally.  I like to use soapUI which is a great tool.  The problem I ran into was it kept failing saying with the error BadContextToken from WCF.  Some time on google showed this is a common problem with SoapUI connecting to WCF with the UserName security type. 
In order to still get my testing done, I ended up creating a second endpoint with its own binding to do testing with SoapUI. Obviously you wouldn't release this for the security hole it creates, but locally and on dev, it allows you nice debugging of the service with SoapUI.
1. Create the new endpoint (next to the old one)
					address ="soapService"
					binding ="wsHttpBinding"
					address ="soapUIService"
					binding ="wsHttpBinding"
2. Then Add the new binding configuration for the new endpoint.
<binding name="tcpWithMessageSecurity">
          <security mode="Message">
            <message clientCredentialType="UserName" />
        <binding name="tcpWithMessageNoSecurity">
          <security mode="None">
            <transport clientCredentialType="None" />
            <message establishSecurityContext="false" />
3. Fix SoapUI 
Now you can configure SoapUI, bind to the wsdl as you would. In the request, click the WS-A tab at the bottom and check the add default wsa:action and Add default wsa:To boxes. 
4. Profit!