Log in

Previous Entry | Next Entry

Customs Cargo System Disaster

The Customs Cargo Management Reengineering (CMR) Project dates back to 1996. In the outsourcing contract, provision was made for the redevelopment of the legacy systems implemented in the mid-1980s. These had been intended to last for ten years (the time that their predecessors had operated for) but it was now certain that they would continue to operate for far longer. While the technology was old (ten years is a very long time in IT), it was far from clear that any great benefit would accrue for merely upgrading it. Instead, the IT branch considered the possibility of "reengineering" -an important buzzword in the mid-1990s. By this point the entire Imports-Exports (MX) system was fully automated. Which is to say, that the computer now did the tasks that were once done by hand, more or less in the same manner as they were done by hand. For example, the system allocated entry numbers in a continuous ascending sequence in the manner of the rotating stamp that had once been used. For a computer system, this required a bit of cunning to prevent locking. At the end of the day, it recorded the last number used. It was now clear that actaul way in which things were done, particularly with respect to the procedures involving the release of goods from Customs control, could now be greatly simplified. This was reengineering. The problem with the term was that to the IT Branch reeningeering meant changing MX, while the MX Branch thought that it merely meant changing IT.

Provision for CMR was incorporated into the contract with EDS that was signed at the end of 1997 and it was given a budget of $40M. It soon hit a snag. Melanie Chalice rejected the plan in the winning bid, on the grounds that a Unix solution was proposed, requesting a new one based around an IBM mainframe. EDS duly drew up a new proposal based on this but submitted a higher cost estimate. Chalice insisted on the same price as the Unix proposal. EDS did its sums over and over and concluded that this was not feasible. Customs therefore put it out to an open tender. Advice from the Gartner Group indicated that EDS was asking too much per function point. However, Customs' estimate of the number of function points in the project was considered to be far too low.

In 2001, this was won by a consortium led by Computer Associates, with Kaz, IOCORE and NCR for applications, IBM for professional services (and some hardware and software under its arrangement with Customs outsource partner EDS), BeTrusted (now Cybertrust) for PKI software and services for the Customs Connect Facility (CCF) "gateway", Novell for identity management and directory services software, and VeriSign for GateKeeper. Costs were now estimated at around $60M. The contract Computer Associates contained severe penalties for any changes to the project specification and Customs began making numerous changes almost immediately. The contract therefore became de jure a cost plus fixed fee one rather than a fixed price one. The initial project plan looked ominous. Owing to the short time frame, it contained many start points but the finish points were clustered around the end of the project in March 2003. If any component ran behind schedule, then the whole project would be behind and this fact might not be known until too late. The project was also extremely ambitious from a technological point of view. Its nearest equivalent, the American ACE system, is so far estimated to have cost more than $US2B and is still far from complete.
ntegrated Cargo System (ICS)

The cornerstone of CMR was ICS, an integrated system giving enhanced risk assessment at the border and allowing more efficient cargo tracking. Its software suite has 23,000 function points. This application was based on an IBM OS/390 mainframe running z/OS with transactions in a CICS environment with DB2 database management. MQ-series provides the mainframe interfaces with the CCF gateway and other business applications. Thus, its core technology was actually older than the systems it sought to replace.

Customs' Web-based user interface, Customs Interactive (CI) was a WebSphere Java application server front end. CI system software was hosted on Unix machines managed as part of the CCF gateway. Transaction application code to support the cargo management business rules for both EDI and CI channels was developed in the AdvantageGen/CoolGen language, in keeping with the philosophy of using 4GLs. ICS's transaction and event processing architectures create and manage events to prioritise and balance message loads across the system to maintain throughput, with automatic exception and recovery management. The design specification consisted of 19,000 pages of analysis for ICS including 800 screens, 16,000 business rules, 70 complex business messages, 850 database tables, 3700 executable load modules, 1800 CICS transaction types, 55 batch jobs, 90 reports and 35 system interfaces.

The Customs Connect Facility (CCF) would be the gateway to Customs' business applications. Importers, exports and brokers can transact via an interactive mode (Customs Interactive) using industry standard Web services or with batch mode EDI. A data transformation facility translated Customs and industry-agreed standards for data exchanges (ie the UN/EDIFACT format used by the existing applications) to Customs' application requirements, significantly reducing customers' previous need to use a plethora of data formats. It also allowed Customs staff to track messages CI ran on Sun Solaris Unix platforms and Cisco routers, with validation and transformation processed on IBM P- and SP-series Unix platforms and Wintel servers running IBM AIX, Win2K, DB2 , WebSphere, Tivoli WebSeal and Baltimore's FormSecure. Overall, the CMR architecture was designed to be multi-tiered, highly available, scalable and to have shared security components with common code bases (for services such as authentication and authorisation). The CCF solution has its origins in the IBM e-business infrastructure reference architecture with J2EE, WS-Security (SAML), XML, UN/EDIFACT D99B and LDAP.

The CMR project was implemented in stages. The first stage was a pilot that connected the ICS, which handles risk assessment and reporting of imported and exported cargo, to a small number of express carriers. This was implemented in April 2003. The second stage involved the redevelopment of the Exports system. For some reason, CMR continued to belied that Exports represented 60% of the total system. Actually, it was more like 6%. It ran overtime and overbudget and, beset by many technical problems, was eventually implemented in October 2004. By now, Customs was admitting to a system that cost over $120M. Customs CEO, Lionel Woodward was informed that early retirement was not an option and that he would have to remain at his post until the system was implemented.

The third stage of the CMR project is the implementation of the more complex and important Imports system. Imports traffic makes up 90 per cent of Customs transactions and there were industry concerns that the new system could buckle under the load when it went online, especially in the run-up to Christmas, the busiest time of the year. For that reason, the system was originally scheduled for implementation in March. This deadline was actually set in legislation which had to be amended for delays in implementation. Delays forced it to be postponed to 1 July. On 24 May 2005 the Minister for Justice and Customs announced that the Government had agreed to an extension to the transition phase of the ICS import system in response to industry’s concerns that, owing to the late release of the system for acceptance testing, insufficient time was available for them to test their own software, adapt business practices and train staff. The legislation was therefore amended - for the seventh time. At this point, the cost of the system was over $200M and the import cut-over was now set for 12 October 2005, this being calculated as the latest possibly date before the Christmas rush. Any delay would now require an implementation date in 2006.

However, after several very quiet weeks in September, Compile (scheduled to be switched off in March 2006, by which time it would celebrate its 20th Birthday) processed a record number of transactions in the first week of October. Perhaps because of the looming deadline, the Christmas rush had arrived early in 2005. On 11 October, there was an eighth postponement, for two weeks. Chaos was already brewing. The sytem had 304 registed bugs affecting almost every component of the system, which was now costed at $250M. The time to get entries through the system was too great; import processing slowed to a crawl and cargo began piling up at the airports and container depots. This meant big money: holding an average-sized container ship at anchor cost about $US45,000 ($A60,000) a day. Warehouse fees are also very high. Some thirteen ships were turned away from Sydney. Melbourne (the country's largest port), Brisbane and Fremantle were also badly affected. With the system more or less down, Customs staff reverted to manual processing, for which staff numbers were completely inadeqate and industrial action started looking like a real possibility. The Minister for Customs, Chris Ellison was called upon to answer questions. A decision was taken on 21 October to persist with the new system rather than scrapping it but special steps were taken to reduce the backlog, the most significant of which was increased use of Compile.


( 1 comment — Leave a comment )
27th Oct, 2005 02:02 (UTC)


( 1 comment — Leave a comment )