Funny Little Story About a Scrum “Implementation”

Funny little story about an “abstract” (if you will), implementation of Scrum that I told to Ken Schwaber at his CSM class in Boston a few years back.

My wife and I, like a lot of parents of young children, were trying desperately one night to get our three year old daughter to eat all her dinner. We tried everything from cutting up her food into smaller and smaller bites to introducing various incentives like ice-cream and TV after dinner .. no matter what the tactic, we just couldn’t get her to eat and were met with all sorts of resistance. At one point, I even made myself more food so that I could eat it along with her. Nothing was working. It was starting to get later in the evening and we knew we couldn’t put her to bed without dinner and I needed to get prepared for the next day for our team’s morning Scrum.

I think maybe because this was on my mind, I decided as a last resort to try some very basic agile approaches to get our daughter to eat. Instead of telling her that she needed to eat her dinner or - insert various parenting text-book consequences here -, I decided to ask her “…how many bites would you like to have?”. Immediately, I had her attention and she responded without much hesitation … “zero”. Well, zero obviously wasn’t going to be acceptable so we negotiated a higher number and finally arrived at two I believe it was. Hey, two was better than zero and meant at least some progress. She agreed and actually began and finished her two bites and then looked to me for feedback on her recent accomplishment. I thought, “Great! Well, how many would you like to have now”. “Three”, she said, and so we proceeded along just like this until her whole plate was finished. My wife, couldn’t believe her eyes, nor anyone else that we demonstrated our strategy too. Scrum Master or Parent Coach? … I thought … hmmm.

Ken Schwaber made a great point to me after I told him this story. A very fundamental part of this success was the fact that my daughter actually felt she owned the task of eating her dinner which differentiated it from all the other day-to-day requests we make to our children. She really felt good about doing it.

Scrum works, even for three year olds.

SVN Relocate Command

svn switch --relocate http://old_svnserver/svn/repo/trunk http://new_svnserver/svn/repo/trunk .

What is your idea of a successful project characterised by?

I’ll extrapolate some of the ‘objective’ and ’subjective’ points for this discussion. I think it’s important to recognize and acknowledge these two categories of criteria in order to maintain and grow your organizations promises as they pertain to it’s offerings. Objectively - meeting deadlines, completing under budget, (completing at all or in some cases, stopping at the right time), low defect rates, customer financial ROI are all objective type criteria that “contribute” to the success story - if you will. Subjectively - the people factors such as raw emotions pertaining to satisfaction, confidence, delightment, excitement and happiness are equally if not more important than the objective criterias and also contribute to the success story. Scoring high in both these categories, which grows your success story, is critical.

The question beyond ‘my idea of what a successful project is characterized by’ then becomes - how do you measure and properly interpret ’success’ from what you know at the end of the project in terms of money spent, time taken etc. as well as what everyone involved thinks and feels about the project, how it went and what had been delivered? It’s important to be able to tell a good story at the very end.”

Insert Spaces Between Capital Letters in a String

Works well with formatting output of form field keys.

preg_replace(’/(\w+)([A-Z])/U’, ‘\\1 \\2′, $string);

Randomize HTML Snippets

This one is very useful for randomizing html snippets such as image tags or just plain quotes.

/** Randomizes the order of the html in the $html array below.
* Don't forget the comma (,) at the end of each line (minus the last line) when adding more items!!
**/

$numItems = 2; // Change this value to add or decrease the number of elements in the $html array to be displayed.

$html = array (

'<img src="/images/imageOne.jpg" alt="" />' ,
'<img src="/images/imageTwo.jpg" alt="" />' ,
'<img src="/images/imageThree.jpg" alt="" />'

);

srand((float)microtime() * 1000000);
shuffle($html);
$i=0;
foreach ($html as $output)
{
if($i&lt;$numItems)
echo $output;
$i++;
}

First / Last Letter of a Column

Will return if you have a record beginning with a distinct letter. sort them and count the records.

Useful for alphabetical lists.

For Left (First character):
SELECT DISTINCT LEFT(title. 1) AS letter. COUNT(*) as count FROM table GROUP BY letter ORDER BY count DESC;

For Left (Last character):
SELECT DISTINCT RIGHT(title. 1) AS letter. COUNT(*) as count FROM movies GROUP BY letter ORDER BY count DESC;

(Remember MySQL is optimized for count(*). count(*) actually being faster than count(id))

Form Post Hijacking

Great article about Form Post Hijacking I found at http://www.anders.com/projects/sysadmin/formPostHijacking/

AKA: Email Form Header Injection
The Problem.
Spammers are constantly being blacklisted and kicked off of networks. Because of this. tricking a non-spamming website into sending spam has become a high priority. One way for spammers to find vulnerable webservers is to test for CGI applications that would allow the spammer to enslave the webserver. Once a vulnerable webserver is found. the spammer can mask the true source of his spam while the enslaved webserver does the bulk of the work.

How do they do that?
A common task websites do is send an email to the owner of the website with whatever data someone has entered into a form. For example. one such script that does this is called formmail.pl or formmail.cgi from Mat’s Script Archive. In this script (as well as many others like it) some fields in the form are used directly in the header of an email. (for example. the Reply-To: field in the email is sometimes set to whatever the user of the form entered in the field called “email” so the owner of the website can easily hit “Reply” to that email and send a response.) If these fields are included unmodified. a spammer can simply overwrite the remaning header lines and effectivly submit any email they wish to through the underlying email system. effectivly enslaving the webserver / email system to send spam.

How exactly do they exploit the script?
The destination of an email is set in its headers. Headers. as everything else in an email. are just lines of text. What seperates the headers from the body of an email are just two blank lines. If your formmail script places anything in the header of an email that is unmodified from what the web user entered. they could easily add those two blank lines. This. of course. would just truncate the headers early and make the body of the email contain some of the headers as well. However. if the web user decides to throw in a few more headers before sending the two blank lines. the underlying email system will listen to those as well. So what the spammers are doing is including a “Bcc:” list of spam victims to the email. When the email subsystem gets the email. it blindly follows what is written in the headers and happily sends one copy of the message to each person listed in the “Bcc:” line. Now of course spammers will probably also add their own subject line and some spam content to the email.

In order to find vulnerable webservers to prey on. spammers usually test the form by sending a sample through that is Bcc’d to and email address they have access to. Usually this is some throw-away address such as a hijacked AOL address. Webmasters are usually alerted to this when they see 5 to 10 trial emails in usually less than one second. They Google the address and hopefully find a page like this one which explains what is going on.

So what can be done about it?
The simplest way to mitigate the danger is to disallow any linefeed or carriage return characters in fields used in email headers therefore disabeling an attacker’s ability to add those two blank lines and trick your mail system into sending whatever they want. Then the problem becomes one of cleaning up a little annoyance rather than being enslaved to do a spammer’s bidding.

Huh? How do I do that?
Well. this is where it becomes a little complicated. The answer to that question depends on what software the form processor on your webserver is using. If you didn’t write your own form processor. your first move is probably to go ask the people that helped you set up your website. Most common form mailers have had bug fixes released since this vulnerability first came out. Have your web hosting provider update their form post code.

If you know what you are doing. check all fields in forms that are used in email headers and strip out the carriage return (\r) and line feed (\n) characters. In perl. this is done like this:

$field =~ s/\r/ /g; $field =~ s/\n/ /g;
If you are using PHP. you can do this for each variable used in email headers:

$_POST['email'] = preg_replace(”/\r/”. “”. $_POST['email']); $_POST['email'] = preg_replace(”/\n/”. “”. $_POST['email']);
How do I know if I’ve been hacked?
As most form to email scripts don’t write down what they are doing in a file somewhere. the best way to tell if your setup has been enslaved to send spam is to check your mailserver logs. If you are with a web hosting provider. they can usually check the logs for you.

What should I do if I’ve been hacked?
You should update your form mailer scripts so continuing attempts are not successful. (do this as described above) The you might want to chase down the throw-away spammer email address used to test for the vulnerability. Most I have seen to date have been addresses at AOL. so you should complain to abuse at aol.com or. as this is a Terms of Service violation. to tosgeneral at aol.com. Many people have already done so with limited effect. so don’t expect AOL to come swooping in and fix everything. but its at least covering the bases.

OK. so I’m clean. Now how do I stop the annoying test emails?
Again. go ask your hosting provider. However. if you have to make the edits on your own. find some fingerprint in the email that flags it as obviously bad. Like if the email field contains the characters “Bcc:”. chances are you are looking at a probe for vulnerability. Just put an “if” statement around the block of code that sends you an email that tests for that. Don’t forget to ignore case so “Bcc:” and “bcc:” are caught. Alternativly you could just look for the two blank lines. however you should be carefull about “\r\n\r\n” and “\n\n” which both will be interpreted as two blank lines. If you are going to do it this way. I suggest you kill all “\r” characters and then search for “\n\n”.

What else can be done?
A better (though more complicated) way to get around all of this is to require something that is easy for a human but hard for a computer to do. One such method that is quite popular is to use a Captcha image with some text in it that the user must enter before submitting the form. Personally. I don’t like the terrible user experience captcha imposes so I favor an image based recognition requirement to submit the form such as the one at the top-right on this page. As you reload the page. notice the images change. The downside with this method is that it takes some substantial server-side programming. which again is heavily dependant on your particular setup. However if you head over to JustHumans.com. you can find a hosted version of this system. Just sign up for an account and generate some JavaScript code to replace your submit button.

If you have other comments. post them on my blog post covering form spam. Link to this page if you think it has been helpfull so others can find it.

Create a New User Group and Assign Content Permissions

The requirement for client site was to have two user groups that are not natively part of the Joomla core access designated group protected content items. Here’s how I did it.

First the SQL to change the database to add the new user groups:

[mysql]

SET @parent_name = ‘Public Frontend’;
SET @new_name = ‘Agents’;

– Select the parent node to insert after
SELECT @ins_id := group_id. @ins_lft := lft. @ins_rgt := rgt
FROM jos_core_acl_aro_groups
WHERE name = @parent_name;
SELECT @new_id := MAX(group_id) + 1 FROM jos_core_acl_aro_groups;

– Make room for the new node
UPDATE jos_core_acl_aro_groups SET rgt=rgt+2 WHERE rgt>=@ins_rgt;
UPDATE jos_core_acl_aro_groups SET lft=lft+2 WHERE lft>@ins_rgt;

– Insert the new node
INSERT INTO jos_core_acl_aro_groups (group_id.parent_id.name.lft.rgt)
VALUES (@new_id.@ins_id.@new_name.@ins_rgt.@ins_rgt+1);

[/mysql]

THE FILES:

/includes/joomla.php

Wrote a new function to check the users actual group id:

/*********************** CHECK USERs GROUP ID - TD Mar 30 /07 ************/
function getUsersGroupId()
{
global $database. $my;
$accessQuery = "SELECT a.id ".
"FROM jos_groups a. jos_users b. jos_core_acl_aro_groups c ".
"WHERE c.group_id = b.gid ".
"AND c.name = a.name ".
"AND b.id = ".$my->id;

$database->setQuery($accessQuery);
$database->query();
$accessRes = $database->loadResult();

return $accessRes;
}
/********************* END CHECK USERs GROUP ID *************************/

Then I changed up the mosMenuCheck function slightly to use the above function to get the users actual group id and return that instead of the gid when logged in. Here’s what I did just below the globals declaration:

/******* Get users group id for content item permission checking **********/
$ugid = "";
$ugid = getUsersGroupId();
/**************************************************************************/

and then instead of “return ($access <= $gid)” I did this:

// Return the users group id or the gid depending if logged in or not - TD - Mar 30 / 07
if($ugid > 0)
return $ugid;
else
return ($access <= $gid);

/modules/mod_mainmenu.php

I have a bit of a customized menu class here originally from the JPL project so basically what I did here was to get the class to check access based on the users actual group id if they or logged in or the gid when they are not. Basically the SQL that pulls together the menu links should be checking access (AND access <= “.(int)$my->gid;). I simply added the condition where if I can get the users actual group id from a function similar to getUsersGroupId() then use the value that the function returns in the AND access clause in the SQL. Here’s exactly what that looks like:
$accessRes being what the function returns. In this case in the menu class. I simply wrote the function code not as a function but within the code.

if(!$accessRes)
$secondLvlQry .= "\n AND access <= " . (int) $my->gid;
else
$secondLvlQry .= "\n AND access IN (0.".$accessRes.")";

/components/com_content/content.php

In the showItem function I did the following right below the globals declarations:

// *******************Added to check users group id - TD Mar 30 / 07 **************
global $my;
if($my->id)
{
$accessRes = "";
$accessRes = getUsersGroupId();
}
//*********************************************************************************//

Then I changed up the main SQL query just slightly in the AND clause that checks access:

// If result from access query above then - TD //
if(!$accessRes)
$query .= "\n AND a.access <= " . (int) $gid;
else
$query .= "\n AND a.access IN (0.".$accessRes.")";

In the show function. I commented out the following:

/*
if ( $row->access > $gid )
{
if ( $noauth )
{
mosNotAuth();
return;
}
else
{
if ( !( $params->get( 'intro_only' ) ) )
{
mosNotAuth();
return;
}
}
}*/

Using an SSL Certificate with Joomla

The simplest way of using Joomla 1.0.8+ with an SSL certificate is to run your entire site over https. To do this. you will need to add redirects from your http pages to the https versions. and make sure that the $mosConfig_live_site value in your configuration.php file contains your https URL.

However. that is not a very satisfactory solution not least because running your site over https is noticeably slower than http.  Typically. you will want the Joomla administrator to run over https. and the front-end website to run over http - perhaps switching to https when a user logs in.  Joomla 1.0.x versions do not support this behaviour natively - to get it working. it is necessary to make some small hacks to certain Joomla files…

The following instructions are based on Joomla 1.0.10.  For other versions. the same principles apply. but there may be slight differences.  Always make sure you are running the latest stable version of Joomla.  Back everything up before you start!

To make sure the Joomla administrator always runs over https…

In administrator/index.php. immediately after the line that says

define( ‘_VALID_MOS’. 1);

Add the following:

// PART 1 - See if user is connecting via SSL
if ($_SERVER["SERVER_PORT"] == “443″ && $_SERVER['SERVER_NAME'] != “localhost”)
{
// reset site config var to SSL equiv
$mosConfig_live_site = str_replace(”http://”. “https://”. $mosConfig_live_site);
}

//PART 2 - Redirect to https if accessed over http (except when running locally)
if ($_SERVER['SERVER_NAME'] != “localhost”)
{
$port = $_SERVER["SERVER_PORT"];
//echo $port;die;
$ssl_port = “443″;  //Change 443 to whatever port you use for https (443 is the default and will work in most cases)
if ($port != $ssl_port)
{
$host = $_SERVER["HTTP_HOST"];
$uri = $_SERVER["REQUEST_URI"];
header(”Location: https://$host$uri”);
}
}

Also add the above code to /administrator/index2.php - immediately after the require_once directives near the start.

This forces the administrator to always use https. However. if you are using IE and find that it keeps warning you about insecure items on the page. you will have to add the code from part 1 to the end of your configuration.php file instead.  Note however. that the code you add there will be lost whenever you save the ‘Global Configuration’ page in Joomla (so you will either have to re-add it after saving. or preferably make changes directly in configuration.php instead of using the Joomla ‘Global Configuration’ screen).

To allow the front end of your website to use https as well as http. open Joomla’s main index.php file. and find the following code (near the start of the file):

define( ‘_VALID_MOS’. 1 );

include_once( ‘globals.php’ );
require_once( ‘configuration.php’ );
require_once( ‘includes/joomla.php’ );

Paste the following after those require_once commands:

// See if user is connecting via SSL
if ($_SERVER["SERVER_PORT"] == “443″ && $_SERVER['SERVER_NAME'] != “localhost”)
{
// reset site config var to SSL equiv
$mosConfig_live_site = str_replace(”http://”. “https://”. $mosConfig_live_site);
}

(Note: As mentioned above. you could add the above code to the end of your configuration.php file instead. However. that will break whenever you change any configuration settings in Joomla. so it is better to put it in the index.php file which is never modified by Joomla - unless you also want to run the adminisrator over https and are finding that you keep getting warning messages about the page being insecure - in which case you will need to add the above to the end of configuration.php instead of index.php as described above).

This ensures that if a request comes in over https. all content is served over https (including images. etc.). In addition. in order to avoid problems with session cookies while switching from http to https. you will need to edit a line in the includes/joomla.php file.  Find the line (round about line 896) that says:

return md5( ’site’ . $mainframe->getCfg( ‘live_site’ ) );

…and replace it with the following:

if (strpos($mainframe->getCfg(’live_site’). ‘http://localhost’) !== false) {return md5( ’site’ . $mainframe->getCfg( ‘live_site’ ) );} else {return md5( ’site’ . str_replace(”http://”. “”. str_replace(”https://”. “”. $mainframe->getCfg( ‘live_site’ ))) );}

These hacks enable Joomla to be able to handle both http and https.  They do not however cause your site to automatically switch to https.  There are 2 ways to acheive this:

1) Instead of using the login module. use the login component. and link to it from a menu item of type URL - specifying https in the URL.  For example. you could create a menu item that points to https://www.yourdomain.com/index.php?option=com_login - that way the entire login process is handled using https.

2) If you want to keep the login module. the login form itself will not be shown over https. however by making the following alterations. the login submission is still protected as the form will be submitted over https when the user clicks on login. In other words. it is just as secure as using option 1. but the user will not see a padlock icon until after they have logged in.

In either case. you need to make a small amendment to the login form.  If you are using option 1. the form to be altered is in the components/com_login/login.html.php file.  If using option 2. it is in modules/mod_login.php.  There is nothing stopping you using both methods. in which case you will need to alter both files.

In modules/mod_login.php. look for the line that says:

<form action=”<?php echo sefRelToAbs( ‘index.php’ ); ?>” method=”post” name=”login” >

Make sure you get the login one. not the logout one (ie. make sure it ‘index.php’. not ‘index.php?option=logout’ in the middle bit).  Replace that line with the following:

<form action=”<?php echo strpos($mainframe->getCfg(’live_site’). ‘http://localhost’) !== false ? sefRelToAbs( ‘index.php’ ) : sefRelToAbs( str_replace(’http://’. ‘https://’. $mosConfig_live_site) . ‘/index.php’ ); ?>” method=”post” name=”login” >

NOTE: If using the login module. you will also need to go into the module parameters (login to Joomla administrator. go to Modules->Site Modules. and click on the login form). and set the login url to the https:// version of the page you want users to be directed to when they log in. If you don’t do this. IE will might you a nasty warning message when you try to log in. and you could be redirected back to http.

You can also optionally redirect to http when the user logs out by setting the logout url in the module parameters. However. IE will display a warning when redirecting to http like that. To get the site back to http when logging out without the pesky warning in IE. you need to change the logout form as well.  Look for the line (nearer the start of modules/mod_login.php) that says:

<form action=”<?php echo sefRelToAbs( ‘index.php?option=logout’ ); ?>” method=”post” name=”logout”>

Make sure you get the one that says ‘index.php?option=logout’ in the middle.  Replace it with:

<form action=”<?php echo sefRelToAbs( str_replace(’https://’. ‘http://’. $mosConfig_live_site) . ‘/index.php?option=logout’ ); ?>” method=”post” name=”logout”>

Firefox may give you a warning when logging out to let you know you are going back to http - the above will not prevent this.

To update the login component. go to components/com_login/login.html.php. find the line (quite near the start) that says:

<form action=”<?php echo sefRelToAbs( ‘index.php?option=login’ ); ?>” method=”post” name=”login” id=”login”>

Replace it with:

<form action=”<?php global $mosConfig_live_site; echo strpos($mainframe->getCfg(’live_site’). ‘http://localhost’) !== false ? sefRelToAbs( ‘index.php?option=login’ ) : sefRelToAbs( str_replace(’http://’. ‘https://’. $mosConfig_live_site) . ‘/index.php?option=login’ ); ?>” method=”post” name=”login” id=”login”>

If you want to revert back to http when they log out. scroll down to the logoutpage function (near the end of the file). and find the line that says:

<form action=”<?php echo sefRelToAbs( ‘index.php?option=logout’ ); ?>” method=”post” name=”login” id=”login”>

Replace it with:

<form action=”<?php global $mosConfig_live_site; echo sefRelToAbs( str_replace(’https://’. ‘http://’. $mosConfig_live_site) . ‘/index.php?option=logout’ ); ?>” method=”post” name=”login” id=”login”>

Joomla (version 1.5) has full https support without any hacks being necessary!

Zip Command

The following examples illustrate typical uses of the command zip for packaging a set of files into an “archive” file. also called “zip file”. The command uses the standard zip file format. The archive files can therefore be used to tranfer files and directories between commonly used operating systems.

zip archivefile1 doc1 doc2 doc3 This command creates a file “archivefile1.zip” which contains a copy of the files doc1. doc2. and doc3. located in the current directory.

zip archivefile1 * This command creates a file “archivefile1.zip” which contains a copy of all files in the current directory in compressed form.

However. files whose name starts with a “.” are not included. The extension “.zip” is added by the program.

zip archivefile1 .* * This version includes the files that start with a dot. But subdirectories are still not included.

zip -r archivefile1 . This copies the current directory. including all subdirectories into the archive file.

zip -r archivefile2 papers This copies the directory “papers”. located in the current directory. into “archivefile2.zip”.

zip -r archivefile3 /home/joe/papers This copies the directory “/home/joe/papers” into “archivefile3.zip”. Since in this case the absolute path is given. it doesn’t matter what the current directory is. except that the zip file will be created there.
The command unzip extracts the files from the zip file.

unzip archivefile1.zip This writes the files extracted from “archivefile1.zip” to the current directory.

Mount Points

Below are various commands for discovering. mounting and un-mounting drives.

Command:
fdisk -l
will give you the hd* (* = number) of the Hard Drive you want to format

To format the drive. as ROOT
enter this command
mkfs /dev/hd*

Line for the /etc/fstab file:
/dev/sda1 /mnt/sdb1 auto noauto.user.owner 0 0

Mount the drive:
mount -t auto /dev/sdb1 /mnt/dev_backups