.jpg)
Revamping Data Management in the Oil & Gas Sector with Azure Data Engineering Solution
Business Problem/Scope of Work
Goal: Enhance student achievement in public primary and secondary schools by implementing an online assessment and administration framework.
The Educational Services faced challenges in managing the transfer of candidate registration and test data, including responses, scores, and other related information to other vendor systems for a series of standardized assessments administered to public primary and secondary schools in the states of Nebraska and Alaska. These challenges included manual data entry, delays in the assessment process, and difficulty integrating different systems used to retrieve test metadata, candidate data, and test session data.
To address these issues, Pronix developed an online assessment and administration framework to automate the data transfer process, improve efficiency, and provide easy access to all relevant data in one place. The goal of the project was to improve the overall assessment process and provide more accurate results for students in Nebraska and Alaska.
Business Solution
The Educational Services company developed a Candidate Data Transfers framework to address the challenges faced in managing the transfer of candidate registration and test data. The framework aimed to improve the overall efficiency of the assessment process by providing a comprehensive solution that streamlines the data transfer process.
The framework featured a process with multiple phases and steps, allowing each phase to be configured at the level of the test program.
It also supported multiple test programs within the same deployment, had the ability to schedule unattended automated execution, provided internal web monitoring and control UIs, used APIs exposed by other components, limited direct access to other component databases, and processed multiple candidates simultaneously, to expedite the transferring process.
Technical Solution
Here's a step-by-step technical solution for the implementation of the Candidate Data Transfers framework for Educational Testing Services using Amazon Web Services:
Deploy Java applications to AWS using existing application deployment tools and processes or integrating them with automated deployment tools and services like AWS CodeDeploy or AWS OpsWorks using Amazon Elastic Compute Cloud (EC2) instances.
Use Amazon Simple Storage Service (S3) to store the candidate admin records files due to its high durability, ensuring the safety and security of the candidate records.
Use Amazon Simple Queue Service (SQS) as a message queuing service to enable distribution of load in CDT BAT instances and dynamically provision optimal quantity and type of compute resources based on the volume and specific resource requirements of the batch jobs submitted.
Use spring batch job and partitioning (using Partitioner and PartitionHandlers) for horizontal scaling and future scalability instead of remote chunking. Use a spring task executor at one node to execute steps in parallel.
Store project-dependent property and value pairs within the database in JSON format, and use a Groovy utility class for validation of the entire XML with XSD.
Use the asyncTaskExecutor of the Spring framework for executing processes to process and write out to the CRDS interface.
This technical solution provides scalability, manageability, and efficiency for client's data transfer process, ensuring high availability and security for the candidate data.
Technologies/Skills Used
We chose Java 1.8 as the primary programming language for the Candidate Data Transfers framework, with Groovy being used for specific functionalities.
The Spring framework (Core, Web Services, Batch) was used for building the web application and web services. Apache Common Chain, Hibernate (4.x and 5.x), Log4j and EhCache were also utilized to facilitate the development of the application.
To manage the project, we used Maven for project management, Jenkins for continuous integration, SVN for version control, JIRA for project management, and Confluence for team collaboration.
We selected Postgresql as the database for storing and managing the candidate records.
The framework was deployed on AWS (Amazon Web Services), and we made use of S3 and SQS for storing and managing data.
The operating systems used were Cent OS and RedHat Enterprise Linux, which provided a stable and reliable environment for running the framework.
This technical solution enabled the Educational Testing Services company to manage data effectively and efficiently, while also providing high availability and security for the candidate data. By leveraging these technologies, we were able to deliver a solution that met the unique needs of the company and their customers.
Customer Success Outcomes
Pronix Inc. helped the educational testing services company to streamline their data transfer process, resulting in several quantifiable outcomes, such as:
Reduced data transfer time: The new system significantly reduced the time it takes to transfer data, resulting in a 30% reduction in data transfer time.
Improved data accuracy: The system ensures accurate data transfer and eliminates data errors, resulting in a 50% reduction in data errors.
Increased efficiency: The new system increased efficiency by automating the data transfer process, resulting in a 40% reduction in manual efforts.
Cost savings: The new system reduced operational costs by eliminating the need for manual data transfer, resulting in a 25% reduction in operational costs.
In conclusion, the implementation of the Candidate Data Transfers framework for Educational Testing Services resulted in significant improvements in several key areas. The ability to onboard new customers, ease of maintenance using AWS monitoring services, and improved performance serving 190K concurrent users led to the expansion of the company's customer base, improved ease of maintenance, and better service for students. These outcomes demonstrate the effectiveness of the technical solution and the benefits that it brought to both the company and its customers.
Latest Case Studies
Our Case Studies
.jpg)
.jpg)
.jpg)
.jpg)
.jpg)
.jpg)
.jpg)
.jpg)
.jpg)
.jpg)
.jpg)
.jpg)
.jpg)
.png)
.png)
.jpg)
.jpg)
.png)
.png)
.png)
.png)
.png)
.png)
.jpg)
.jpg)
.jpg)
.jpg)
.jpg)
