n order to measure project performances, the quality assurance department of an organization releases baselines periodically. These may be variance of actual effort over planned effort, the variance of actual schedule over-planned schedule, organizational productivity ratio, defect density, etc.,
Baselines are released every quarter or periodically. An improvement in adherence to schedule or a reduction in the defect density would imply that the organization has made improvements. For example the release may state that the organizational baseline I for schedule variance is 8.9 percent and the organizational baseline II for schedule variance is 5 percent. The conclusions may hence be that the organization has improved its schedule adherence by 3.9 percent or in other words the projects are delivering to the customers on time.
Now let us understand that such an analysis has been done on two different data sets at two different times. One data set measured the effort,schedule and other project metrics during Quarter I, the second data set measured the effort,schedule, and other project metrics during Quarter II. The mean may have significantly shifted, or the variance may have shifted or in other words, the comparison may not be being made on similar data.
For example, during Q1 the company may be executing large scale projects and during Q2 the company may be executing medium scale projects. In the case of large projects,estimates tend to have more variances and so the mean of all schedule variances hence has lowered.
Similarly, the variability between individual data points in the distribution may be significantly different due to data collection methods. During Q1 the baseline may have used a manual data collection process and during Q2 the baseline may have used data entered in the automated time management system by the employees.
When conclusions regarding baselines are derived, it is essential to see if there is a significant change to parameters which are correlated to the metric being evaluated for process improvement. More specifically, a measure such as schedule variance is correlated with many parameters such as complexity of the project, training level of the employees, work timings etc., When a result related to improvement in the performance of the process is obtained, then one should make sure that the factors on which the metric is correlated do not change significantly.
For example a mean schedule variance of 9 percent where the mean size of the projects are 20 FP should not be compared to a mean schedule variance of 5.9 percent with mean project size 10 FP or a project or a mean schedule variance of 9 percent with employees with low skill should not be compared to projects with a mean schedule variance of 5.9 percent that have employees of high skill.
In such cases one should use a statistical regression equation which expresses schedule as a function of many other variables to scale the measurements to a common base.