This page describes the 7th release of the EGI Quality Criteria (QC), that is used for verification of software products. These QC drive the Software Verification in EGI Software Provision workflow.
QC_DOC_1
Products must provide release notes with:
Verifier must assure that the document exists and contains the requested information.
Include a link to the document in the report.
QC_DOC_2
Products with client tools must provide documentation on its usage
Verifier must assure that the document exists. For any detected flaws in the document, open a ticket with the issue.
Include a link to the document in the report.
QC_DOC_3
If the product provides a public API, it must be documented. The documentation should cover all the existing public functionality of the API. If the product implements a well-known or standard API (e.g. SRM), any missing functionality must be documented.
Verifier must assure that the API is documented.
Include a link to the document in the report.
QC_DOC_4
Products must provide an administrator guide describing installation, configuration and operation of the system. The document should contain the following information:
Verifier must assure that the document exists. For any detected flaws in the document, open a ticket with the issue.
Include a link to the document in the report.
QC_DOC_5
Products must have a compatible license for using them in the EGI Infrastructure. For Open Source products, compatible licenses are those accepted by the Open Source Initiative and categorized as Popular and widely used or with strong communities:
Other licenses accepted by the Open Source Initiative and listed as Special Purpose are compatible with the infrastructure (when applicable):
Any other license, and non Open Source products will be evaluated by the verification team in coordination with the Operations Community.
License of the product must be one of the listed before.
Include the license type in the report.
QC_DIST_1
Products must be available in the native packaging format of the supported platform (RPM or DEB packages).
Packages must install without issues in a machine configured without any external repositories (valid repositories are the standard OS repo, UMD repo and EPEL repo for RH based distros)
Packages must follow the OS policies (name of packages, use of filesystem hierarchy, init scripts, ...). For any detected issue, open a ticket.
Packages must be signed (or the repository where they are fetched from is signed for Debian-based distros)
yum/apt-get log can be included as appendix to the report.
QC_UPGRADE_1
Non-major updates should be installable in a machine configured with the previous version.
Verifier should update a working product with the verified version. The update should not cause any problems.
yum/apt-get log can be included as appendix to the report.
QC_SEC_1
Primary authentication token within the infrastructure is the X.509 certificate and its proxy derivatives.
Verifier must use X.509 certificates as authentication token. The appliance should also support proxy derivatives (e.g. VOMS proxies).
QC_SEC_2
Certificates and proxies generated with SHA-2 cryptographic hash funtions must be supported by the middleware
Verification must be performed with SHA-2 certificates for the service and for clients.
See EGI QC wiki for more information on SHA-2 certificates.
QC_SEC_3
All middelware using proxies must accept RFC proxies as credential tokens for authentication
Verification must be performed using a RFC proxy. Such proxy can be obtained with the -rfc option of voms-proxy-init.
See EGI QC wiki for more information on RFC proxies.
QC_SEC_4
All services requiring authorization should be able to use an external ARGUS service that manages the authorization policies and provides mapping of accounts.
Verifier configures an external ARGUS service for the product being verified. Test a banning policy and a allowed policy to the resource. Check that the policies are enforced.
See EGI QC wiki for using the ARGUS service of the verification testbed for this test.
QC_SEC_5
World-writable files and directories are dangerous since they allows anyone to modify them, several vulnerabilities in recent years have been due to world writable files and directories being present when they should not be. No product should include or create world writable files in order to prevent new vulnerabilities being introduced in the future.
Verifier must assure that the product does not include or creates world-writable files or directories. If any world-writable files are required for the normal operation of the service, these should be documented by the Techonology Provider, a ticket reporting the files must be opened. Logs and config files must not be world-writable.
See EGI QC wiki for example test.
Proposed by the EGI SVG RAT to prevent new vulnerabilities in the future.
QC_SEC_6
If the product uses configuration files with any kind of passwords, those files must not be world readable.
Verifier must check the default permission of configuration files that contain passwords. None of them must be readable.
QC_INFO_1
Resource information exchanged in the EGI Infrastructure must conform to GlueSchema v1.3
Verifier must run glue validator to check GlueSchema 1.3 conformance.
See EGI QC wiki for information on running the validator and list of allowed issues.
QC_INFO_2
Resource information exchanged in the EGI Infrastructure must conform to GlueSchema v2.0
Verifier must run glue validator to check GlueSchema 2 conformance.
See EGI QC wiki for information on running the validator and list of allowed issues.
QC_INFO_3
Resource information published by Information Discovery Appliances must include the middleware version of the middleware
Verifier runs a query to check the middleware version.
See EGI QC wiki for information on example queries.
QC_MON_1
Services must be monitored by the monitoring infrastructure of EGI by using the public inerface of the service. The exact tests to perform for each service are determined by the operations community. The product should work with the existing probes for the service.
TP should provide a (or a set of) monitoring probe that tests that the service provides the expected functionality correctly that runs integrated in the monitoring infrastructure of EGI.
If the product is already being tested in the verification testbed, check that the tests pass OK. If the tests do not pass clarify the source of the problem (if it's a limitation of the verification testbed, the criterion passes as WARNING).
If it's not being tested, contact the verifiers mailing list to configure the Nagios machine.
If there there are no tests for the product, do also contact the verifiers mailing list to decide on the result of the verification.
QC_ACC_1
Job Execution Appliances must generate accounting records for the resource consumption of the users. The exact format of the accounting records is defined by the operations community.
These records must be transmitted to the EGI central database periodically.
Execute some jobs and force the generation and transmission of records. Check that there are as many records as job sumitted and that the transmission was started.
QC_SUPPORT_1
Technology Providers must enrol in GGUS as 3rd level support for the products verified by the Quality Assurance team of EGI. Any further integration with TP-specific bug tracking software is entirely up to the Technology Provider.
Pass if Technology Provider enlisted as 3rd level support in GGUS.
QC_FUNC_1
Verification must include basic functionality tests of the product that assures that it performs its intended functionality.
Verifier must configure the product and start its services in the verification testbed. If the verifier is not able to start the services, the TP should be contacted to clarify any issues, opening GGUS tickets if neccesary.
Once the product is installed and running, the verifier should perform basic tests to assert that its basic functionality is met. The quantity and level of detail of the tests can be adjusted for each release (i.e. major releases should include more testing than minor or revision releases.
See EGI QC wiki for test examples for each product.
Include in the report:
If the product cannot be operated because of some missing functionality or major bugs, set the product as rejected.
QC_FUNC_2
Verification should check (whenever feasible) any new features and bug fixes included in the release.
Check the release notes of the products and test the relevant features and bug fixes included in the release (if they are feasible to be tested within the verification testbed). Take into account that the product will be also tested in Staged Rollout, if there are changes that should be tested in a production environment notify state it clearly in the report.
Include in the report: