Passwordless SSH between two machines using certificates

I’ve been meaning to post this somewhere useful for a while now, so here it is.

Assumption

Machine A wants to connect with SSH without passwords (with cert) to Machine B

Method

On Machine A:

ssh-keygen

Make sure you pick no pass phrase.

scp .ssh/id_rsa.pub user@machineb:.ssh/id_rsa.tmp

On Machine B:


cat id_rsa.tmp >> authorized_keys
chmod 600 authorized_keys

Connect as normal from machine A to machine B

Unable to browse network shares on Windows 8

Again a small side note for anyone experiencing this problem, accessing network shares on windows 8 with elevated privileges seems to be disabled in Windows 8 (not sure about 8.1?), to get around this problem make the following registry changes:

  1. Locate HKEY_LOCAL_MACHINE/SOFTWARE/Microsoft/Windows/CurrentVersion/Policies/System
  2. Create a new DWORD entry with the name EnableLinkedConnections and value 1.
  3. Restart your computer

You should now be able to view the shares.

QED

Errors when running large numbers of unit tests via mstest on a windows server (just a note really)

This is really just a note for future reference and to help anyone else experiencing this issue:

C# projects with hundreds of unit tests (more than a thousand or so) having unit tests failing randomly with the following error:

The system could not find the environment option that was entered

or

Not enough storage is available to process this command

I battled to find the solution but this KB from Microsoft solved the problem:

http://support.microsoft.com/?scid=kb%3Ben-us%3B285089&x=15&y=11

Essentially it involved the following (from the article in case the link goes dead at some point):

 

The IRPStackSize parameter specifies the number of stack locations in I/O request packets (IRPs) that are used by Windows 2000 Server, by Windows Server 2003, and by Windows XP. You may have to increase this number for certain transports, for media access control (MAC) drivers, or for file system drivers. Each stack uses 36 bytes of memory for each receive buffer. This value is set in the following registry subkey:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters

The default value of the IRPStackSize parameter is 15. The range is from 11 (0xb hexadecimal) through 50 (0×32 hexadecimal).

If the subkey is not present, you can use Registry Editor to add the subkey. To do this, follow these steps:

  1. Click Start, click Run, type regedit, and then click OK.
  2. Locate and then click the following registry subkey:
  3. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters
  4. Click Edit, point to New, and then click DWORD Value.
  5. Type IRPStackSize, and then press ENTER to name the value.Note Type IRPStackSize exactly as it is displayed. The value name is case sensitive.
  6. Click Edit, and then click Modify.
  7. In the Data Value box, type the value that is appropriate for the network, and then click OK.

 

Hope this proves useful to someone in the future, sure took me a long time to find the solution.

Accuracy and Precision

Today I got asked what I thought was the difference between accuracy and precision as it pertains to relative estimation, I wasn’t ready for the question though and it’s been sitting on my mind bugging me so here are my thoughts:

I would think accuracy would be how close the estimation was to the truth and precision is how often you predict the same estimation given the same or similar complexity, in other words how repeatable you are. Accuracy is how right or wrong you are and precision is about how consistent you are (irrespective of right or wrong).

So, if you correctly estimate over and over, you’d be accurate and precise but if you estimate incorrectly, but are consistent about the “incorrectness” then you would be precise but not accurate. If you estimate correctly only every now and then for the same given complexity then you would be accurate but lack precision.

Interesting to hear what others think (like if I’ve lost my mind completely :) )…

 

Continuous Improvement

Thinking about continuous improvement the other day I tried to figure out what makes a continuous improvement process work and what makes it stick, these are my thoughts:

  • It should be a slow, gradual process of change, no “sudden movements”.
  • It should be a continuous flow of change, no settling down into a rut.
  • It should be a culture, not a bunch of individuals with ideas.
  • It should be unanimously accepted.
  • It should be measurable.
  • It should be celebrated.
  • It should be evangelized.
  • There should be room for failure, and for success.
  • It should be rewarded.
  • It should be encouraged.
  • It should become default behaviour.
  • It should be transparent.

What else?

 

Emailing attachments directly to a Dropbox folder…

I’ve been toying with this idea for a while now, wanting to email attachments to my Dropbox account into a specific folder. There are lots of canned solutions to do this, but most of them require you to trust some third party with your documents. If you’re ok with that check out this post.

I decided my paranoia was high and I would not trust a third party. To achieve autonomous Dropbox emailing I figured I’d need to do the following:

  1. Install Dropbox on my headless Ubuntu server.
  2. Create an email account to receive the email attachments (let’s say dropbox@mydomain.com).
  3. Modify the postfix installation to redirect mail to dropbox@mydomain.com to a PHP script.
  4. Create a PHP script to process the inbound email stream and strip out all attachments saving them to a folder.

Install Dropbox

Lots of links for this one out there, not going to rehash it but the basics are:

  1. Download the latest Dropbox.
  2. Run it.
  3. Use the link supplied to link your account to your server.
  4. Create a service to run Dropbox at start.
  5. Do an initial sync.

You can follow the instructions here, or Google for it.

Create Email Account

I created a user called dropbox  and ran the install above as that user. One note if you follow the link, I did not do the auto startup as a cron job but as a standard service in /etc/init.d

Modify Postfix

I created a new transport in master.cf:


dropbox unix - n n - 50 pipe
flags=R user=dropbox argv=/usr/bin/php /usr/local/bin/dropboxattach/dropbox.php -o SENDER=${sender} -m USER=${user} EXTENSION=${extension}

then I added an entry to my main.cf to define different transport options:

transport_maps = hash:/etc/postfix/transport

Now edit /etc/postfix/transport and add an entry for your email address receiving the dropbox attachments:

dropbox@mydomain.com  dropbox:

then run

postmap transport 

to (re)build the transport database

Finally reload postfix

postfix reload 

The PHP Script

I found this nifty little Mime Mail Parser script at http://code.google.com/p/php-mime-mail-parser/, download the two files and add it to your code source. Here is the script I created to intercept the mail and parse for attachments. Check comments for explanation.


<?php

require_once('MimeMailParser.class.php');

// phrase to check for in SUBJECT to make sure only valid emails are processed
$check = "Add to dropbox";

// location to save attachments to
$save_dir = "/home/dropbox/Dropbox/attachments";

// temporary file name for incoming mail message
$file = "/tmp/mail.message." . microtime(true);

// delete the temp file if it exists, just in case
if (file_exists($file)) {
unlink($file);
}

// export the mail message to the temporary file
$data = file_get_contents("php://stdin");
file_put_contents($file, $data);

// parse the contents of the message and save attachments
$Parser = new MimeMailParser();
$Parser->setPath($file);

// get the subject and check for valid "phrase"
$subject = $Parser->getHeader('subject');

if (!($subject == $check)) {
// delete the temp mail message
if (file_exists($file)) {
unlink($file);
}
exit();
}

// get all attachments and save to directory
$attachments = $Parser->getAttachments();
foreach($attachments as $attachment) {
// get the attachment name
$filename = $attachment->filename;
// write the file to the directory you want to save it in
if ($fp = fopen($save_dir.$filename, 'w')) {
while($bytes = $attachment->read()) {
fwrite($fp, $bytes);
}
fclose($fp);
}
}

// delete the temp file (probably redundant but just in case)
if (file_exists($file)) {
unlink($file);
}

?>

For some “security” I added a “catch phrase” that must be the subject as a kind of “authentication”. Remember to save the script into the same directory as the Mail Parser code and everything should work.

That was it, I can now email dropbox@mydomain.com and it appears in my dropbox folder Attachments .. AWESOME!

A great team

For me agile is about many things but first and foremost it is about people and dynamics between people, you get the right recipe going and anything is possible. I work with what I think is a great team, but what makes them a great team? I’m not entirely sure but here are some of my ideas of traits of a great team (in no particular order):

  • There’s a good level of respect not only for each other as people, but in each other’s ability to do stuff.
  • They trust each other.
  • The team has built in redundancy so anyone can take time off at any time, no silo’d experts.
  • There’s lots of laughter and healthy tomfoolery, they know when to work, but they also know when to play (at work!).
  • They allow failure and learn from it,  focusing on the problem, not the person.
  • High levels of collaboration, both at work and at play. They help each other out constantly, no-one is ever “too busy” to help.
  • They’re cross functional, there’s a good mix of skills amongst the team, helping with redundancy.
  • Everyone feels they can talk openly about issues and problems.
  • There is no fear (of failure, of specific people), they take chances and enjoy challenges.
  • They self organize, there is no one leader, they all lead, they all follow.

There’s more I’m sure, but those are the ones I can think of right now. At the core of it I think people must feel like they belong, that they are respected for their skills and trusted with responsibility and above all treated like people, not resources.

Thoughts?

Velocity for the layman

WARNING:This is a brain dump of an idea I’ve been playing around with, some confusion may occur!

As a ScrumMaster I often struggle to explain the concept of velocity to people, a critical concept to grasp to understand how to report on progress and capability. I often get these questions thrown at me:

  • How many features can we complete in X weeks? or…
  • How many bugs can one developer fix per day ?
The danger of these questions, for me, is not so much in the question (which doesn’t make sense) but in what happens afterwards:
  • 1 developer = 1 story per sprint so 2 developers = 2 stories per sprint, right?
  • 1 developer can fix one bug in 1 day, we could double our rate by doubling our people!! or…
 So, to the challenge: how to explain the danger is making all pieces of work equal and how to explain the benefits and logical sense of measuring in velocity.
Let’s start with the analogy: You’re going on holiday, you need to travel from city A to city B (your goal) with 5 stops between.
Doing it wrong would be to assume that you multiply the time to the first stop by the number of remaining stops and that would be your final travelling time.
Doing it right would be to look at the distance between each stop, calculate the average speed you can maintain, and then add up the travel time between each stop. Even better would be to:
  • Adapt your average speed based on environmental conditions. Bad weather, day/night travelling, fatigue, additional rest stops, travelling through mountains? These all affect your average speed.
  • Keep track of your average speed as you travel, this should give you an indicator of exactly what average you can maintain.
  • Check what affects the distance between towns like road closures, alternate routes.
  • Be ready to adapt to changing conditions, like accidents and sudden road closures.

Given the above you should be able to accurately predict when you will arrive at your goal. As a bonus you can also work backwards. Given your need to arrive at a particular time, you can sum up the total distance and divide it by the time you have and this should give you the speed you should try to average.

So, to bring it back to velocity, you average travel time can be equated to the teams average velocity (or capability). Stops are user stories/bugs and distance between cities is the effort of doing the story. Conditions of road, road closures, mountain travelling, predicted bad weather etc is your complexity.

Here’s hoping this analogy makes sense, if it doesn’t please let me know. I still think it’s missing something though so I may add to it at some point…

When will we be finished?

I’m a firm believer that all work should have an agreed end date, but end dates are always a bone of contention and cause endless confusion and wasted arguments and disagreements. These usually hover around the “when will we be finished?” question and, more to the point, no properly defined (empirical) way of working it out.

That in mind I thought I’d lay down the process we use to arrive at our end date. I will also attempt to do this without using the word scrum or agile, not because I don’t like or agree with them, but because those words tend to create predefined thoughts and have a life of their own (unfortunately).

 

So first the basics: In a nutshell we work in iterations, releasing something (hopefully of value) every 2 weeks based on a prioritized backlog. The backlog is constantly updated as stories are added / removed / completed.

Now for the fun. In order to calculate an end date with reasonable empirical accuracy you need the following: remaining effort and an average effort per sprint / day the team is capable of.

Effort Scale

Effort is difficult to define but key to the entire process. Everyone should agree and understand what is meant when we say “X amount of effort”. What you must realize is that it does not matter what unit of measure you use to define your “effort” as long as:

  • it is consistently used across all stories
  • it is understood by everyone
  • something of effort X closely matches another thing of effort X,  for small values of X. 
  • the units should increase roughly exponentially. This is to allow for indication of greater uncertainty with significantly higher numbers. Some people use the Fibonacci sequence (1, 2, 3, 5, 8, 13 …). Also remember to allow for zero effort.

We struggled to settle on our effort scale but eventually had to settle on a time relative scale (for various reasons):

  • Very Easy (1 point) – roughly a day to complete
  • Easy (2 points) – between a day and 2
  • Medium (4 points)  - roughly half a sprint
  • Hard (7 points) – roughly a full sprint
  • Epic (20 points) – unknown or way more than a sprint

and of course 0 points for quickies.

Effort per story

Assigning effort to stories is an exercise the entire team should do but failing that at least the majority of the team. Stories should also be continuously rechecked to see if the effort is still correct, especially stories with high effort assigned to them.

The process we use is as follows:

  1. Discuss the story
  2. Vote on an effort, if there’s dispute, use the highest of the choices, if there’s huge uncertainty, assign the highest effort.
  3. Rinse and repeat

Remember also to compare stories of similar effort. Another way or form of checksumming is to, during planning, assign tasks to the story (as in what you’re going to do to achieve the story) and then assign a rough hour estimate to do the tasks. Sum up the hours for the story and an equivalent effort story should probably have a similar amount of hours, although this is only really useful for well understood stories.

Average Effort Per Iteration

Calculating the effort per iteration (sometimes called the velocity) is a matter of adding all effort units (points in our case) for all completed stories for a sprint. It’s important to only tally up completed stories or the whole process is pointless. Average effort is just the sum of the effort for X number of iterations divided by X. Obviously the more data you have the better. What we also do is remove the highest and lowest values (as anomalies) from the calculation, to account for those strange iterations (over Christmas, Easter etc.)

Remaining Effort

Remaining effort is simply the sum of all effort that has not been completed.

The Formula

Calculating the end date then becomes the following:

By Day: date today + (remaining effort / average velocity / number of work days in iteration) divided by 5 and then multiplied 7 to account for weekends = your end date

By Iteration: current iteration number + (remaining effort / average velocity)  rounded up to whole number = final iteration of work, end of iteration is end date

and that’s it, you’ve predicted a future date based on historical data using a reasonably empirical method. You can improve the accuracy of this estimation by doing the following on a regular basis:

  • Re-evaluate the backlog from an effort point of view, focusing particularly on high effort stories.
  • Recalculate your average velocity at the end of each sprint and factor it in.
  • Factor in holidays and leave into your calculation.

Comments or suggestions welcome

gSouthAfrica 2.0 Day 1

Ok, so gSouthAfrica Day 1 didn’t really grab my interest in the slightest, perhaps the coolest thing I saw was some of the features around Google+ that are either just released (Google Ripples) or planned soon (like the integration with YouTube to add music play lists and share and control these from Plus, although can’t really see a use case for a built in music player in Plus?). Anyway, so besides the funky Plus things coming the rest of Day 1 was pretty much a rehash of last year. The keynote again focused on Google’s plans for expanding into Africa revolving around solving some of the key problems to introducing wide scale technology to the continent. I must admit they are making some nice strides to solving this but are a far way from making Google an everyday name in the dark reaches of Africa.

All in all for me the day was boring, but then all the technologies they spoke about I know on a deeper level already so I expected that. Day 2 (Developer Day) sounds like it could be good though (at least I hope so) and I’m actually looking forward to it…

It's all about improvement