Test&Go Uses Python for Data Validation
|Category:||Software Development, Government|
|Keywords:||Testing, Document Management, Quality, Administration|
|Title:||Test&Go Uses Python for Data Validation|
|Summary:||Test&Go, a data validation product from QualiControl, used Python to solve data quality problems in a French government agency.|
When one of the French Government Agencies was looking for a way to overcome their growing data quality problems, they chose to use the Test&Go software suite from QualiControl, along with associated consultancy, to help them develop a solution.
The problem was the following: every year, over a period of one month, more than 800,000 French companies sent their employee year-end reports in electronic form. More than 35% of these reports had errors or exceptions, and reports highlighting these exceptions had to be sent back to the originating company for correction of the data, before again being sent back to the Agency. This process was repeated until the data received was free of any critical errors, causing the whole submission-validation-correction cycle to last up to six months.
In order to resolve this problem, the Agency had several options to choose between. These options included writing their own application, outsourcing the problem, or using a market solution. After evaluating the cost and development times of each, they decided to pick Test&Go as a combined solution for the following reasons:
The Test&Go Integrated Development Environment allowed for very rapid development and testing of a solution. Visual programming in Test&Go made it very intuitive and efficient to use.
Test&Go used Python as a scripting language. The choice of Python as a development language initially seemed a risky one to the Agency, but its fast learning curve, tremendous power, supportive user community and broad range of books on the subject, quickly convinced them it was the right choice.
The initial script development was done as a joint development between the Agency and QualiControl. The project was done in this way to help the Agency develop a solution in the shortest time possible, with simultaneous training and consulting. This resulted in a script containing over 940 standard Test&Go validation controls, enhanced with more than 5,000 lines of Python code. Total development and testing was completed in approximately one month, instead of the six months that would have been required if taking a traditional development approach.
The Agency decided to deploy a stand-alone script to 40,000 of the companies that had encountered problems the previous year.
The implementation of the Test&Go script met with enormous success. The following year the rejection rate for the companies submitting data, validated using Test&Go, was reduced from 35% to 5%, and the total processing time was halved to three months. The rejection rate could not be reduced further at the time as not all validations could be done at source, i.e. on the client site, because access was required to certain internal databases at the Agency.
Following a very successful pilot phase the Agency is currently deploying its script using the eTest&Go Server and Web Services technologies, allowing them to reach even more companies, and further reduce the rejection rate and processing times.