Limecraft is often used in mission critical environments, consequently quality assurance of software development and operations are hugely important. This includes software development and operations of the SaaS platform, on premise instances, as well as of the complementary installable software (Limecraft Edge, Limecraft-Tools and Limecraft Cloud Connector) for manipulation of locally stored audiovisual material.


TABLE OF CONTENTS

 


Quality assurance in the software development process

  • Procedures and systems are available to support peer-review of code such that validation is performed on complex code changes that could introduce regressions or unwanted side effects;
  • Automated software builds can be reproduced without human intervention, minimising the risk of making mistakes;
  • Automated unit testing of individual components to ensure basic functionality is operational and no regressions are introduced.
  • Automated integration tests for complex software such as Limecraft Edge and Limecraft Cloud Connector agents, supported by manual test sessions described by test scenarios to verify proper workings of new functionalities.

 

Project Management

Quality management at the level of project management of the software development process is also implemented in such a way that software issues (created either internally, or triggered from the customer support system) are properly tracked (using Atlassian JIRA) through their entire life cycle from creation to resolving and deployment. The planning of effort done to resolve these issues is done using a digital planning tool that is integrated with the issue tracking software. Finally, specifications for software development are documented by means of an Atlassian Confluence WIKI system, which in turn is also integrated with the issue tracking system.


Delivery of bespoke Development

Custom development product delivery (i.e. new features or improvements outside the regular product roadmap or that require an extensive installation or integration procedure) are handled using first a definition of work statement including functional and non-functional requirements (listing also component deliveries to be made by all involved parties and dependencies of the elements), along with acceptance criteria and acceptance testing plan. After this definition phase, the required work is transferred to the issue tracking system, where the further development and delivery is managed (cf. 2/).

 

Deployment of the SaaS systems

  • A lengthy phase in “development” in which features are deployed in a fully working and feature-complete internal SaaS testing environment. This development environment includes all components of the production environment, minus the redundancy setups. As such, functionality can be tested in a real-life scenario, including possible side-effects of new code on existing components.
  • For some features which require extensive testing by customers before adoption in production systems, Limecraft provides an isolated but publicly accessible ‘labs’ pre-production environment.
  • When features are completed and tested by the customer in the labs environment, they are deployed to a staging environment, which mimics the production environment with a similar redundancy setup.
  • Finally, when deployment to staging completes successfully, the SaaS software is deployed to the production environment in a fully automated fashion such that no human intervention is required and mistakes are avoided. This deployment process is further supported by a maintenance log for interventions such that inadvertent actions (if they would occur after all) can be traced and reverted.

 

Documentation

All components of the system are well-documented on three levels: 

  • low-level reference API documentation of internal use and third party developers, 
  • intermediate level documentation which explains concepts, interactions, etc. to help developers quickly understand the workings of the system, and 
  • highest-level introductions to customers and developers to validate potential workflows and functionality before starting implementation. 

Documentation is an important aid in helping quality assurance in the development process: the system capabilities are better understood and developers and users spend less time guessing functionality and more time following recommended practices.


Release Management

As part of the continued commitment to our quality assurance procedures, Limecraft uses an automated release management system. Limecraft users are notified which fixes and features will be deployed.  It provides customers with clear insights about the planned changes of the SaaS platform and on premise installations, for both the overall product stack as for any bespoke developments.


Preventive and Corrective Measures

Limecraft also implemented procedures for preventative and corrective measures. This comprises the following:

  • Vital parts of all SaaS environments are actively live tested, monitored and configured with alerting. This encompasses the gathering of live metrics on low-level components (CPU, memory, etc) and application-level metrics (DB performance, ingest speed, etc.), but also live testing of end-to-end ingest functionality of the platform every 30 secs.
  • As mentioned above, a continuous improvement procedure is also in place to constantly evaluate the current state of quality assurance of software and services delivered by the company, the result of which leads to preventative improvements before actual problems arise.
  • Finally, an active customer feedback loop is in place at Limecraft. As a preventative measure, customer input is continuously recorded and taken into account for improvements of the product. Furthermore, feedback is also sought during the development of features to ensure a proper and industry-approved implementation, and to gauge potential impact and regressions on existing processes that needs to be considered when the development is underway.

 

Support Processes

Regarding procedures for addressing deviations, complaints and escalation processes, we refer to the general terms of service or to your specific support services agreement which may be in place, in which the support processes are further explained. 


Issues or incidents are managed by support agents. Any applicable resolution is taken into consideration as part of the software development and deployment processes, as explained above. Whenever issues are closed in development, this resolution is also fed back to the support system for formal customer follow-up. Depending on the severity of the reported issue, a report will be provided on root causes and future mitigations to be put in place to avoid repeats of the incident. In addition, for nonconformity incidents, e.g., software defects or outages with a wider impact, a post-mortem report will be distributed to all (potentially) impacted customers.

 

Auditing of the Quality Management System

The quality management system is audited in two ways:

  • At the management level, we have implemented a bi-yearly review of the quality management procedures. In this review, an extensive analysis of the mentioned procedures is executed, and improvements are defined. These improvements are then planned and executed as part of the software development process (which can include work on (integrations with) supporting systems (issue tracking, customer documentation, planning tools etc.), or they are taken up as updated or new procedures as part of the daily work followed by employees.
  • Additionally, the quality assurance processes are also continuously evaluated and reviewed on a smaller scale during the regular software development, management and deployment process. Sufficient room is made for developer, operations engineer and customer support input into this process such that issues that surface are addressed timely and properly.