Case Study | Data Analytics & Insights | Performance Specialists | Spike95

Case study: Data analytics & insights – Ensuring timely delivery of critical metrics

A major retailer approached Spike95 for help with their data & analytics platforms following a major Oracle Retail implementation.
 
Our challenge was to “get data moving” and resolve a number of issues along a complex data journey from the data lake through flattening, transformation, data warehousing and out to a MicroStrategy reporting solution.

Benefits:

  • Built a repeatable data generation and loading framework.
  • Identified regression and performance issues allowing customer to change and tune end to end load of sales data through to EDW within required 4 hour overnight window including:
    • Identified optimum ingestion batch size and reduced data flatteningtime by 50%.
    • Determined ideal cost and performance configuration for ETL.
    • Tuned Microsoft SQL Data Warehouse (Synapse) for most efficient use of capacity.
    • Identified SQL Server configuration and maintenance improvements which reduced cube load time on the reporting server.
  • Found critical bottlenecks before go live:
    • Hardware constraint with 3rd party reporting software.
    • Limitations with in-house developed “unzip service”.
    • Inefficiencies with database maintenance tasks which had a significant impact on data publish times at the MicroStrategy reporting layer.

BACKGROUND

Spike95 led the analysis, design and implementation of a number of data initiatives, working within the client’s data division and the wider Oracle Retail programme.

  • Some real data had been presented but not in sufficient and reliable enough volume. Previous data creation efforts were not to the volume or variance required for full testing.
  • The client’s teams were working with the client’s central performance team to build execution capability and adopt a core performance framework. Spike95 were previously involved in this work and were therefore able to enhance these efforts.
  • Getting reliable volumes of data into Data Lake was key so that this could then flow into other layers.
  • Waiting for real data was no longer practical as it would not allow the projects to deliver on time.
  • Key goal was to generate synthetic data in volume until it can be replaced by real data starting with Data Lake and iterating through each layer.

OUR APPROACH

Our performance experts drove the planning of key performance activities including:

  • An independent evaluation of current methods used to data creation in depth and confirm the details within this proposal.
  • Confirming readiness to test across all areas.
  • Examining existing tools including an in-house “data exploder” and designing the technical approach to data creation.
  • Investigating data recovery techniques to allow tests to be repeated with confidence.
  • Determine volumetrics for flows from Oracle Retail into the data layers including throughput required and flow rate, response time required to process data and capacity.

Performance test execution then focused on getting data moving:

  • Modelling the messages into Data Lake for key data flows.
  • Creating volumes of data and flowing into Data Lake.
  • Determining how to then flow data from Data Lake > flattening > EDW > reporting.
  • Conducting pattern-based tests for those flows, identifying critical bottlenecks.

CONCLUSION

  • Modelling data flows and creating synthetic data will get data moving and accelerate complex data programmes.
  • Numerous bottlenecks will exist requiring a focused, iterative and forensic approach to testing.
  • Automated tests and scripts are required to allow repeatability. This is both to execute tests and for test environment creation, teardown, reset and data load.
  • Performance testing is essential to ensure load and transformation can occur in a timely manner. It will ensure valid data can be delivered when end users need it.

Download as a PDF

No contact details needed, click below to download a copy of our case study with our compliments. Please consider the environment and read online or print responsibly.

Want to know more?

Some of our other client success stories

Digital Smart City
Large scale data exchange will only benefit users if the data is reliable and the application performs well. Spike95 successfully modelled the data flows and user interactions and tested to ensure performance prior to launch for a major Smart City platform.
LEARN MORE
Global Online Banking
A major bank needed to accelerate the deployment of high-quality change into production for their digital platforms. Spike95 defined and implemented a roadmap of performance improvement activities.
LEARN MORE
Smart Trains
Spike95 accelerated delivery and ensured performance which enabled the launch of some of the UK’s first smart trains.
LEARN MORE
Global Fashion Retail
Spike95 has been working with a leading online global fashion brand since our formation with a team of consultants leading a number of performance initiatives.
LEARN MORE
Port and Border Security
Spike95’s experts implemented the overall testing strategy to enable automation & DevOps capability and tooling for a machine learning and AI platform which progressed items through security checkpoints in high volumes.
LEARN MORE
Online Finance
Spike95 specialists implemented a centralised automation and performance testing service for an online bank to remove their regular service outages and increase the flow of quality change.
LEARN MORE
Public Sector Digital Transformation
Spike95 created two Centres of Excellence covering Test Automation and Performance Testing for UK government agency then managed service roll out across multiple projects.
ALL CASE STUDIES
Retail Remote Testing Delivery
Spike95 improved remote testing delivery with cloud infrastructure, open source tooling and coaching for a major UK fashion retailer.
LEARN MORE
Data Analytics and Insights
Spike95 gave a fashion retailer confidence that data processing performance was sufficient to ensure timely delivery of analytics and insights reporting.
LEARN MORE
Energy and Commodities Trading Software
Spike95's experts reproduced complex deadlock issues in Endur and executed tests to prove tuning solution worked with no issues. Automation of integration testing brought a faster change cadence.
ALL CASE STUDIES
Oracle Retail Transformation
Spike95 drove a number of initiatives for end to end, performance, security and operational acceptance testing of a major Oracle Retail transformation project for global fashion and cosmetic retailer.
ALL CASE STUDIES
Increasing Delivery Cadence for a Critical Government Platform
Our client had a clear, bold vision for their new software platform and embraced DevOps methodology from the outset making automated testing critical to their success.​
LEARN MORE
Previous
Next