Solving the problem of forgetting to take back library books, and having fun doing it…

Problem

With access to 4 library cards, each with 30 book capacity, that’s a whopping 120 books we can take out. The current “reminder” system from our local library involves emailing you 2 days before it’s due, and then 1 week AFTER it’s due (and you’ve picked up fines). We’re currently sitting at around $20 worth of fines and that’s not good at all.

Solution

Throw technology at the problem!

I built an end to end system using a Python, NodeJS and PushBullet. Allowing me to…

  • access a RESTful service to get info on current taken out books, per library card and…
  • send notifications as books approach due date.

Here’s how I did it:

  1. Scrape web site for data, using Python with HTMLParser to log in and extract the latest book data from the website, dumping it as a JSON string.
  2. Create a NodeJS service to wrap the Python data request and expose it as a RESTful service.

Tada… now we get notified via PushBullet (https://www.pushbullet.com/) whenever books are about to become due.

Code available on GitHub

Remember, it ain’t pretty, but it works :)

Pi in a box

I’ve had this long running obsession to create a neat enclosure for the Raspberry Pi’s I have, mainly because they lie around all over the place. My first attempt involved using an old drive case (with built in fan) that ended up looking something like this (open):

Pi in a case

Which worked pretty well, but then I then decided I wanted EVERYTHING boxed. This would mean I would have to enclose two Pi’s, a drive, a USB hub, and a network switch of some kind (I don’t like wireless for the Pi’s).

In the end I wanted something I could plug a single network cable and a single power cable in to, that would replace my existing web of cables. I was also constrained by money so this had to be something I could cobble together relatively cheaply.

Long story short, I’ve ended up with this:

Pi's in a box

 

which is still a work in progress, but close to what I want.

The container is an old wooden wine box (slide in lid not shown, cost: $2). The router is an old DLINK Wireless Router (free), 2 Raspberry Pi’s (model B) screw mounted into the sides of the box (had these already), a USB hub (to power the drive, also free), an old power supply (not even sure where I got this originally, was lying in a box, so free), a female to female network adapter ($1), a DC-DC converter ($12, I splurged on this for obvious reasons), a 1TB HDD (had this already), a 12V fan (free) and lots of cabling (which I custom made to the right length). So cost wide I’m out less than $20, which wasn’t too bad.

The power supply is rated at 12V, 5A (60W), the DC-DC converter drops it to 5V, 5A (25W) and this supplies everything with power (I made sure everything ran 5V). The converter came with 3 outputs (conveniently) and a power switch, a perfect fit for what I wanted to do.

I’ve just built this so whether it’s stable or not is still to be determined. I know Pi’s have issues with power supply so for the next few days I’ll trial this to see if it’s stable. I’m also concerned with the heat inside the box (the Pi’s are running 10 degrees warmer than usual) hence I’m testing the attached fan to see if it remains stable with it running, if so I’ll mount it as an extractor fan, which should help with the heat. I also need to either get or make a shorter USB Type A-B cable for the USB hub to Pi connection, trying to find one but will probably end up “making” one. I then also need to clean up the “hole” I made for the power supply and network point, looks messy at the moment. Finally I’ll need to mount the breakout breadboard I use for accessing the GPIO ports on the Pi, I’ll probably end up attaching that to the lid with a detachable cable.

In the end I was quite pleased with how it ended up, a bit to do still but I like what’s been done so far. Of course I now need to worry where the NEXT Pi will go, when it arrives :)

Some more pics (more available here):

2014-09-02 22.52.312014-09-02 22.52.24

 

 

 

 

 

 

 

 

 

 

Update: I’ve mounted the fan and added a custom (shorter) USB cable. I think I’m pretty much done for now other than pimping the box with decals (and cleaning up the hole I made :) ). Here are updated pics, with the breakout breadboard mounted on the side of the box.

2014-09-03 22.56.40 2014-09-03 23.00.41

 

 

 

 

 

 

 

 

 

 

Update: Almost 2 months after I posted this and it’s running super stable. I think I rebooted one of the Pi’s twice in all that time. I’ve since upgraded the SD cards to a (hopefully) more stable brand, and swapped out the fan for a 12V PC fan. The Pi’s now run at a cool 36-40 degrees constantly, even under load. I might experiment with over-clocking…

Passwordless SSH between two machines using certificates

I’ve been meaning to post this somewhere useful for a while now, so here it is.

Assumption

Machine A wants to connect with SSH without passwords (with cert) to Machine B

Method

On Machine A:

ssh-keygen

Make sure you pick no pass phrase.

scp .ssh/id_rsa.pub user@machineb:.ssh/id_rsa.tmp

On Machine B:


cat id_rsa.tmp >> authorized_keys
chmod 600 authorized_keys

Connect as normal from machine A to machine B

Unable to browse network shares on Windows 8

Again a small side note for anyone experiencing this problem, accessing network shares on windows 8 with elevated privileges seems to be disabled in Windows 8 (not sure about 8.1?), to get around this problem make the following registry changes:

  1. Locate HKEY_LOCAL_MACHINE/SOFTWARE/Microsoft/Windows/CurrentVersion/Policies/System
  2. Create a new DWORD entry with the name EnableLinkedConnections and value 1.
  3. Restart your computer

You should now be able to view the shares.

QED

Errors when running large numbers of unit tests via mstest on a windows server (just a note really)

This is really just a note for future reference and to help anyone else experiencing this issue:

C# projects with hundreds of unit tests (more than a thousand or so) having unit tests failing randomly with the following error:

The system could not find the environment option that was entered

or

Not enough storage is available to process this command

I battled to find the solution but this KB from Microsoft solved the problem:

http://support.microsoft.com/?scid=kb%3Ben-us%3B285089&x=15&y=11

Essentially it involved the following (from the article in case the link goes dead at some point):

 

The IRPStackSize parameter specifies the number of stack locations in I/O request packets (IRPs) that are used by Windows 2000 Server, by Windows Server 2003, and by Windows XP. You may have to increase this number for certain transports, for media access control (MAC) drivers, or for file system drivers. Each stack uses 36 bytes of memory for each receive buffer. This value is set in the following registry subkey:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters

The default value of the IRPStackSize parameter is 15. The range is from 11 (0xb hexadecimal) through 50 (0x32 hexadecimal).

If the subkey is not present, you can use Registry Editor to add the subkey. To do this, follow these steps:

  1. Click Start, click Run, type regedit, and then click OK.
  2. Locate and then click the following registry subkey:
  3. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters
  4. Click Edit, point to New, and then click DWORD Value.
  5. Type IRPStackSize, and then press ENTER to name the value.Note Type IRPStackSize exactly as it is displayed. The value name is case sensitive.
  6. Click Edit, and then click Modify.
  7. In the Data Value box, type the value that is appropriate for the network, and then click OK.

 

Hope this proves useful to someone in the future, sure took me a long time to find the solution.

Accuracy and Precision

Today I got asked what I thought was the difference between accuracy and precision as it pertains to relative estimation, I wasn’t ready for the question though and it’s been sitting on my mind bugging me so here are my thoughts:

I would think accuracy would be how close the estimation was to the truth and precision is how often you predict the same estimation given the same or similar complexity, in other words how repeatable you are. Accuracy is how right or wrong you are and precision is about how consistent you are (irrespective of right or wrong).

So, if you correctly estimate over and over, you’d be accurate and precise but if you estimate incorrectly, but are consistent about the “incorrectness” then you would be precise but not accurate. If you estimate correctly only every now and then for the same given complexity then you would be accurate but lack precision.

Interesting to hear what others think (like if I’ve lost my mind completely :) )…

 

Continuous Improvement

Thinking about continuous improvement the other day I tried to figure out what makes a continuous improvement process work and what makes it stick, these are my thoughts:

  • It should be a slow, gradual process of change, no “sudden movements”.
  • It should be a continuous flow of change, no settling down into a rut.
  • It should be a culture, not a bunch of individuals with ideas.
  • It should be unanimously accepted.
  • It should be measurable.
  • It should be celebrated.
  • It should be evangelized.
  • There should be room for failure, and for success.
  • It should be rewarded.
  • It should be encouraged.
  • It should become default behaviour.
  • It should be transparent.

What else?

 

Emailing attachments directly to a Dropbox folder…

I’ve been toying with this idea for a while now, wanting to email attachments to my Dropbox account into a specific folder. There are lots of canned solutions to do this, but most of them require you to trust some third party with your documents. If you’re ok with that check out this post.

I decided my paranoia was high and I would not trust a third party. To achieve autonomous Dropbox emailing I figured I’d need to do the following:

  1. Install Dropbox on my headless Ubuntu server.
  2. Create an email account to receive the email attachments (let’s say dropbox@mydomain.com).
  3. Modify the postfix installation to redirect mail to dropbox@mydomain.com to a PHP script.
  4. Create a PHP script to process the inbound email stream and strip out all attachments saving them to a folder.

Install Dropbox

Lots of links for this one out there, not going to rehash it but the basics are:

  1. Download the latest Dropbox.
  2. Run it.
  3. Use the link supplied to link your account to your server.
  4. Create a service to run Dropbox at start.
  5. Do an initial sync.

You can follow the instructions here, or Google for it.

Create Email Account

I created a user called dropbox  and ran the install above as that user. One note if you follow the link, I did not do the auto startup as a cron job but as a standard service in /etc/init.d

Modify Postfix

I created a new transport in master.cf:


dropbox unix - n n - 50 pipe
flags=R user=dropbox argv=/usr/bin/php /usr/local/bin/dropboxattach/dropbox.php -o SENDER=${sender} -m USER=${user} EXTENSION=${extension}

then I added an entry to my main.cf to define different transport options:

transport_maps = hash:/etc/postfix/transport

Now edit /etc/postfix/transport and add an entry for your email address receiving the dropbox attachments:

dropbox@mydomain.com  dropbox:

then run

postmap transport 

to (re)build the transport database

Finally reload postfix

postfix reload 

The PHP Script

I found this nifty little Mime Mail Parser script at http://code.google.com/p/php-mime-mail-parser/, download the two files and add it to your code source. Here is the script I created to intercept the mail and parse for attachments. Check comments for explanation.


<?php

require_once('MimeMailParser.class.php');

// phrase to check for in SUBJECT to make sure only valid emails are processed
$check = "Add to dropbox";

// location to save attachments to
$save_dir = "/home/dropbox/Dropbox/attachments";

// temporary file name for incoming mail message
$file = "/tmp/mail.message." . microtime(true);

// delete the temp file if it exists, just in case
if (file_exists($file)) {
unlink($file);
}

// export the mail message to the temporary file
$data = file_get_contents("php://stdin");
file_put_contents($file, $data);

// parse the contents of the message and save attachments
$Parser = new MimeMailParser();
$Parser->setPath($file);

// get the subject and check for valid "phrase"
$subject = $Parser->getHeader('subject');

if (!($subject == $check)) {
// delete the temp mail message
if (file_exists($file)) {
unlink($file);
}
exit();
}

// get all attachments and save to directory
$attachments = $Parser->getAttachments();
foreach($attachments as $attachment) {
// get the attachment name
$filename = $attachment->filename;
// write the file to the directory you want to save it in
if ($fp = fopen($save_dir.$filename, 'w')) {
while($bytes = $attachment->read()) {
fwrite($fp, $bytes);
}
fclose($fp);
}
}

// delete the temp file (probably redundant but just in case)
if (file_exists($file)) {
unlink($file);
}

?>

For some “security” I added a “catch phrase” that must be the subject as a kind of “authentication”. Remember to save the script into the same directory as the Mail Parser code and everything should work.

That was it, I can now email dropbox@mydomain.com and it appears in my dropbox folder Attachments .. AWESOME!

A great team

For me agile is about many things but first and foremost it is about people and dynamics between people, you get the right recipe going and anything is possible. I work with what I think is a great team, but what makes them a great team? I’m not entirely sure but here are some of my ideas of traits of a great team (in no particular order):

  • There’s a good level of respect not only for each other as people, but in each other’s ability to do stuff.
  • They trust each other.
  • The team has built in redundancy so anyone can take time off at any time, no silo’d experts.
  • There’s lots of laughter and healthy tomfoolery, they know when to work, but they also know when to play (at work!).
  • They allow failure and learn from it,  focusing on the problem, not the person.
  • High levels of collaboration, both at work and at play. They help each other out constantly, no-one is ever “too busy” to help.
  • They’re cross functional, there’s a good mix of skills amongst the team, helping with redundancy.
  • Everyone feels they can talk openly about issues and problems.
  • There is no fear (of failure, of specific people), they take chances and enjoy challenges.
  • They self organize, there is no one leader, they all lead, they all follow.

There’s more I’m sure, but those are the ones I can think of right now. At the core of it I think people must feel like they belong, that they are respected for their skills and trusted with responsibility and above all treated like people, not resources.

Thoughts?

Velocity for the layman

WARNING:This is a brain dump of an idea I’ve been playing around with, some confusion may occur!

As a ScrumMaster I often struggle to explain the concept of velocity to people, a critical concept to grasp to understand how to report on progress and capability. I often get these questions thrown at me:

  • How many features can we complete in X weeks? or…
  • How many bugs can one developer fix per day ?
The danger of these questions, for me, is not so much in the question (which doesn’t make sense) but in what happens afterwards:
  • 1 developer = 1 story per sprint so 2 developers = 2 stories per sprint, right?
  • 1 developer can fix one bug in 1 day, we could double our rate by doubling our people!! or…
 So, to the challenge: how to explain the danger is making all pieces of work equal and how to explain the benefits and logical sense of measuring in velocity.
Let’s start with the analogy: You’re going on holiday, you need to travel from city A to city B (your goal) with 5 stops between.
Doing it wrong would be to assume that you multiply the time to the first stop by the number of remaining stops and that would be your final travelling time.
Doing it right would be to look at the distance between each stop, calculate the average speed you can maintain, and then add up the travel time between each stop. Even better would be to:
  • Adapt your average speed based on environmental conditions. Bad weather, day/night travelling, fatigue, additional rest stops, travelling through mountains? These all affect your average speed.
  • Keep track of your average speed as you travel, this should give you an indicator of exactly what average you can maintain.
  • Check what affects the distance between towns like road closures, alternate routes.
  • Be ready to adapt to changing conditions, like accidents and sudden road closures.

Given the above you should be able to accurately predict when you will arrive at your goal. As a bonus you can also work backwards. Given your need to arrive at a particular time, you can sum up the total distance and divide it by the time you have and this should give you the speed you should try to average.

So, to bring it back to velocity, you average travel time can be equated to the teams average velocity (or capability). Stops are user stories/bugs and distance between cities is the effort of doing the story. Conditions of road, road closures, mountain travelling, predicted bad weather etc is your complexity.

Here’s hoping this analogy makes sense, if it doesn’t please let me know. I still think it’s missing something though so I may add to it at some point…

←Older