Scorecard Performance Monitoring (StreetLights)

The Street Light Performance SCORECARD is directly impacted by THREE types of iSLIMS changes:

1. PROBLEM PROFILE CHANGES (located under iSLIMS setup): This includes changes to PERFORMANCE GROUPING, # days allowed for response, # days allowed for quick turn, the percentage number used to evaluate which SCORE (#5 thru #1) must meet.
   
   
2. DAILY WORK TICKET LINE ITEM COMPLETIONS: Work ticket line items are created, and closed daily. When the Score Card report is run, if a work ticket is OPEN for the time frame selected - it is not included into the scorecard result. When the Score Card is run the next day, or weeks later or even a year later -- when a work ticket is CLOSED - (and the line item is associated with a PERFORMANCE GROUP (defined above)) - then it will get included in the ScoreCard result.
   
   As of this writing (10.4.2009) there are OPEN LINE ITEMS in iSLIMS that date back to 2006. When these line items get closed -- it affects HISTORICAL data - because there were created some time back.
   
   Also, batch data changes (such as changing "Defective Fixture" to a different problem, or adding T-Base issues for selected tickets behind the scenes) will further impact these line items. Thus, just when you thought the Score Card was "final" -- as soon as new batched data is implemented - the score card is re-run showing the NEW results.
   
   
3. SOFTWARE CHANGES: Those changes that affect how a DUE DATE for a work order line item is generated or calculated, and those changes over the span of 3 years that affects the calculation and display of Score Card results. The Score Card result has been a collaborative process - and the algorithms CURRENTLY used can be seen via this link.
   

Below is an iSLIMS Time line that summarizes how changes OVER TIME to the above criteria impacts the Score Card Performance Monitoring Report. This will hopefully show why criteria in use TODAY -- is different from PAST YEARS - and thus Score Cards generated in the past will display different values from those generated today.

This timeline is important, because depending on WHEN a Score Card report was generated - the values in the score card get adjusted (fine tuned) or corrected based on DDOT requirements. Thus, once any changes are made that affect the problem profile, daily work order line item completions or software changes - the Score Card will be reflected accordingly to display the new results. It would be correct to interpret that the SCORE CARD has evolved over time - and continues to be FLUID depending on how the variables are modified.

• March 2006: iSLIMS was launched inHOUSE (within the DC intranet) for DDOT and vendor to enter and track repairs based on citizen complaints imported nightly via Hansen
  
  
• June 2006: iSLIMS was launched to the WEB for Street Lights to monitor the repairs of Street Lights by it's vendor. At this time, PERFORMANCE GROUPS (such as PM 2 thru 69) had NOT been defined within iSLIMS. DDOT was working on translating the PM contract requirements ...into values that could be incorporated into iSLIMS. No Scorecard is available as the criteria has not been defined.
  
  
• May 2007: Hansen tickets that were closed by the vendor, are automatically CLOSED. DDOT no longer has to close tickets both in iSLIMS and Hansen
  
  End of Contract Year 1. DDOT attempts to create PERFORMANCE GROUPS for problems - so that a SCORECARD can be developed.
  
  
• July 2007: iSLIMS is modified to include PERFORMANCE GROUPING -- and problems are updated to become associated with these PERFORMANCE GROUPS. A field for PERFORMANCE GROUPING is added to the PROBLEM PROFILE. This allows DDOT to group problems into PM 2 thru 60.
  
  
• August 2007: Draft of SCORECARD is created offline, and made available to DDOT only. DDOT is concerned that performance groups are not yet fully vetted - and is not ready to share the ScoreCard results with Severn. There is also concern that work tickets are still OPEN (especially for those problems that have a longer turnaround) - and thus the numbers for the ScoreCard change, as tickets get closed over time.
  
  DDOT is concerned about the number of work order LINE ITEMS that remain open, even tho the work order itself has been CLOSED. We resolve this by: (a) automatically closing all line items that show outstanding - when the work order itself was closed and (b) modifying the iSLIMS software, so that when a work order gets closed -- all associated line items get closed with it.
  
  
• November 2007: DDOT discovers that Severn is using a problem called "Defective Fixture" that allows 30 days for response, instead of a more common problem with a 5 day turnaround. DDOT requests an automatic update of all work tickets that include problems with "Defective Fixture"....and re-assign these to "Install Fixture" that has a 5 day turnaround. All due dates were modified for these work tickets. This meant that THOSE AFFECTED work tickets that previously displayed a "Defective Fixture" problem ON time, may now report that same problem (changed to "Install Fixture") as being late.
  
  
• December 2007: Severn has expressed concern that whenever they create a new work order LINE ITEM - it creates a DUE DATE based on the original entry date of the work order. For example, if a work ticket is created 12.1.2007.....and on 12.15.2007 Severn adds a new line item...(such as "light out")....then the line item will appear as being due on 12.6.2007....instead of 12.21.2007. Thus, the moment Severn enters a new line item with a short turnaround (in this case) -- it is automatically OVERDUE.
  
  DDOT's position is - that they don't want Severn to linger on repairing the work ticket, and by allowing Severn to create a due date base on 12.15.2007 entry -- would simply give them more time. Thus, DDOT wants to hold Severn accountable to the original dates.
  
  With that said, DDOT requested a software update to iSLIMS - that would allow DDOT to determine if SOME problems (such as "Light Out") could get created based on the DATE ENTERED as opposed to the ORIGINAL work order date. This update was implemented, and allows DDOT to toggle a switch in the PROBLEM PROFILE for this purpose. As of this writing -- all switches in the PROBLEM PROFILE are set to using the original WORK ORDER date to base the due date of any line item entered.
  
  Here is the impact: Using the above example, if the repair turns out to be substantial, then Severn would be better off (in terms of the Score Card) - to simply close out the work order...and create a new one. If Severn were to use the original Work Order, then they are dinged each time for any line item entered -- as being late before they can even make the repair. This may be precisely how DDOT wants this to work -- but it is important to understand the impact on the vendor as well.
  
  
• January, 2008: DDOT continues to revise the PROBLEM PROFILE, and especially the performance grouping. "Photo Cell Hanging" is added to the list, and those problems that are deemed "obsolete" are removed. Extensive analysis is performed to determine which problems SEVERN is most likely to use, and which problems DDOT would like Severn to use. As adjustments are made to the Performance Grouping, and other profile values - the Score Card displays new results. The SCORE CARD continues to be created offline -- allowing DDOT and SAIC to review the results prior to distributing to Severn.
  
  
• March, 2008: The amount of data stored and displayed on iSLIMS Crystal reports begins to create performance problems for iSLIMS. Street Light data prior to March 2006 is archived from the primary database. Still, large reports are giving iSLIMS the blues. It is determined that in order to prevent the iSLIMS server from crashing - we must temporarily limit the number of days in which a report can be run - to 30 days. Thus, DDOT and Severn can run reports -- but only in 30 day segments. The Performance Score Card is now being run on a monthly frequency - tho it still remains OFFLINE - and is provided to DDOT and SAIC for their review.
  
  
• September 2008: Efforts begin to replace CRYSTAL reports (which has limited the reporting timeframe) with a product that allows large reports to be generated, and better number crunching. All iSLIMS reports undergo upgrades to use this new "report generator" -- and the Score Card report is coded to be available ONLINE.
  
  
• December 2008: All iSLIMS reports have been re-coded with the new report generator - and graphs are included on reports to show trends. The Score Card report is now available ONLINE -- allowing ANY user with report access to see the results of the score card at any time.
  
  
• January, 2009: SAIC notes a discrepancy in the algorithm between the OFFLINE score card results, and the ONLINE version. This affected SCORE #4 results, and was modified so that anything with a #4 - was further evaluated for ANY LATE tickets. A new version of iSLIMS with this modification was published - and shared with SAIC/DDOT.
  
  
• September, 2009: No software changes to the Street Lights SCORE CARD have been implemented since January, 2009.
  
  When Score Card results for RECENT HISTORY (past 9 months) are compared on a weekly or monthly basis - there are only 2 FACTORS that would contribute to the display of different values. These 2 factors are: Changes to the PROBLEM PROFILE or CLOSING out open and overdue LINE ITEMS.
  
  It is especially noteworthy, that when RECENT months are reviewed using the MONTHLY SCORE CARD results, these recent months are especially susceptible to work order LINE ITEMS changes. For example, if you were to run a MONTHLY SCORE CARD for a month ending 2 days ago....and then compare the results when run again 15 days later -- it is HIGHLY likely that many of the problems were not complete the first time the report was run. Depending on how TIMELY Severn is in completing work order line items -- you may continue to note different results in the Score Card - as each of these line items eventually gets included onto the report. This is probably why in the original days of the Score Card deployment, DDOT would hold onto the results until numerous months have passed -- to allow work line items to get complete, and reduce the variablity of the Score Card results.
  
  But wait! What about running a Score Card for Contract Year 2....and then a few months later -- why are the numbers changing? As long as there are OPEN and LATE work line items that exist for those years -- and assuming that there are NO changes to the PROBLEM PROFILE or Software changes -- then YES, these numbers will continue to fluctuate. Should any changes to the PROBLEM PROFILE get made as well, then you now have TWO factors that affect the values.
  
  When Score Card that were generated offline for PRIOR contract years are compared with a "NOW" online printout - it will be VERY difficult to achieve identical results. The 3 factors that contribute to the Score Card have evolved extensively over time. This is due to PROFILE CHANGES made over a 3+ year time frame, Work Ticket Line items that are closed NOW, but were not completed at the time of a prior report and software changes that has evolved to refine the algorithms, and to make the Score Card report available anytime, online.

Below is an analysis of the data (as a snapshot of the iSLIMS database on 10/2/2009 at 11:45PM) as it affects the ScoreCard in general, and specifically for PM02:

• PM 02 is composed of the following individual "problem options"
  ?? (a catch all problem item when the problem is not yet known)
  (Install Fixture)
  (Light off/on)
  (Light out)
  (Light out-Defective Fixture)
  (Photo Cell Hanging)
  
  
• Few changes have been made to the Problem PROFILEs since 12.22.2008. A new problem code was entered on 9.25.2009 by J.Blevins for "Repair holes for Crown". As of this writing - this new code is NOT associated with a Performance Group - and thus would not affect the Scorecard. However, it it were to be added at a later date to a performance group - this will alter the work order COUNTS that get applied to the scorecard report.
  
  
• PM02: There are 208 work order line items that are OPEN and LATE in iSLIMS since 2006. This means the work is showing "Not done". My preliminary review of these work tickets - is that much of the information was probably inaccurately entered. For example, a problem may display "Install Fixture" - with comments saying "refer to DDOT", yet the assignment still shows to be Severn. When these tickets are eventually CLOSED - they will get added into the Performance Score Card result based on the ENTRY DATE of the WORK ORDER. Click here for more details. Is Severn really working on these tickets -- or did they simply fall thru the cracks?
  
  
• PM02: There are over 4000 line items since June 3, 2006 that were completed LATE - many of them significantly overdue. These tickets presently appear on Score Card results -- again, based on the ENTRY DATE of the WORK ORDER.
  
  
• ALL Performance Groups Analysis: The last worksheet displays ALL work ticket line items that are LATE. These tickets may be still OPEN (not completed yet) or CLOSED. If they are closed -- the line item is factored into the Score Card. If the ticket is still OPEN -- it does not appear on the scorecard until closed.
  
  

As you can see -- for those work order line items that are EXTENSIVELY late -- as soon as the work is completed -- the historical ScoreCard data is adjusted to reflect the change. This is why any and all history can CHANGE over time.

This is why Severn feels that the reports are inconsistent over time. But the "Data-reality" is that there are MANY OPEN AND LATE WORK ORDER line items - and when these do not get closed out in a timely manner -- they affect historical results and scorecard results at a later date.

Moving forward -- Items for DDOT to consider:

• If a FIXED SCORECARD result (one that won't change over time / and is not dependant on the date the work order was recieved) is preferred over potentially dynamic data - this can be accomplished by altering how dates are being evaluated for the report.
  
  For example, Instead of evaluating the work order line item by the WORK ORDER ENTRY DATE (current method), the line item could be evaluated by the COMPLETION date. Thus, the values in the reports will remain consistent over time. (i.e. only those line items completed in June 2009 -- will appear in the June 2009 Scorecard. A month, or year later -- those results will never vary - because you can't add more work tickets completed in June 2009 - once June 2009 has passed).
  
  
• LOCK DOWN the Problem PROFILE feature. Only 1 DDOT Superuser individual should have access to this feature. The potential for accidental adjustments -- and the impact it could have on Performance Reports is quite risky.
  
  
• DDOT and SEVERN should frequently run the PROBLEM DETAIL report, and choose the LATE option on the Command Center. Choose a date range from 6/2006 to present. This helps to keep an eye on work tickets that start to fall thru the cracks -- before years go by.