Schema Validator counter inconsistent

[SimonRinguette]

http://bpmn-miwg.github.io/bpmn-miwg-tools/

In the result table, Software AG has 10/10, Trisotech has 12/12... we are not certain why Software AG are missing two results.

Also, C.5 and C.6 seems to not be checked.

[falko]

@tstephen could you have a look into this? It seems like not all test results are passed into the schema validation.

[falko]

Software AG has C.1.0 and C.1.1 submitted but for some reason they don't get picked up, neither by the Schema Validation Tool nor any of the other XML-based tools.

C.5 and C.6 are not showing up for any of the submitting tools.

[falko]

Still not showing: http://bpmn-miwg.github.io/bpmn-miwg-tools/overview.html#ARIS%20Architect%2010.0.10 =(

[falko]

@tstephen any update on this?

[falko]

Has the site already been rebuilt? It is still not showing up on the site.

[tstephen]

It will rebuild during the night. Per our conversation on Friday, this 'fix' is speculative rather than proven.

[falko]

Okay, thanks for trying. Let's leave the issue open until we see the results tomorrow.

[falko]

Still not showing. =(

[dgagne]

5 Oct Webmeeting
o Tim will look at fixing the automation, so it runs on new submissions and show proper count
o We also have to validate the count for C test case that seem to not consider new test cases added to that section

[tstephen]

I have fixed the generation of submissions.json but that was bpmn-miwg/bpmn-miwg-test-suite#885 rather than this one.

The reason that the max schema validations varies for different tools lies in the logic of this piece of JavaScript:

bpmn-miwg-tools/index.html

Line 65 in c6c5e9e

max += 1;

Basically it is calculating the max per tool so it is impossible to get a consistent max across all tools. What is coded here should be regarded as a boolean: If count == max then all submissions are schema valid. I'll look at converting it to a tick

[SimonRinguette]

@tstephen thanks for the fix, maybe not related but the image on the miwg page that links to the table seems to be broken:
https://github.com/bpmn-miwg/bpmn-miwg-test-suite under Test results of BPMN 2.0 tools that participated in the tests

[tstephen]

Clarification at meeting 05 Jan 23: It is by design that the Schema Validation column shows the number of files for the vendor (not the reference) as denominator. The issue is that some (eg Aris) appear to have complete submissions judging by other columns but fewer than the maximum 14 in the schema validation column. Hence something to be investigated

[tstephen]

Investigated the errors with findings as follows (click the Schema Valid link in the table to see for yourself):

1. ADONIS: B.1.0: roundtrip, B.2.0: roundtrip, C.1.0: roundtrip and C.6.0: roundtrip all correctly flagged as failing schema validation
2. ARIS is missing C.1.0: roundtrip and C.1.1: roundtrip submissions hence 12/12 not 14/14. The files are present but the encoding of the umlaut on line 433 and 131 respectively makes them unparseable using the validating parser. xmllint explains:

   $ xmllint --schema ~/git/bpmn-miwg-tools-gh-pages/./schema-tool/src/main/resources/schema/BPMN20.xsd ARIS\ 10.0.20/C.1.0-roundtrip.bpmn --noout
   ARIS 10.0.20/C.1.0-roundtrip.bpmn:433: parser error : Input is not proper UTF-8, indicate encoding !
   Bytes: 0xE4 0x72 0x65 0x6E
      <semantic:userTask xmlns:camunda="http://activiti.org/bpmn" name="Rechnung kl
   

3. Signavio has 2 schema invalid files: B.1.0: roundtrip and B.2.0: roundtrip though for some reason B.1.0 reports it to be ok. Again xmllint explains:

   $ xmllint --schema ~/git/bpmn-miwg-tools-gh-pages/./schema-tool/src/main/resources/schema/BPMN20.xsd Signavio\ Process\ Editor\ 10.0.0/B.1.0-roundtrip.bpmn --noout
   Signavio%20Process%20Editor%2010.0.0/B.1.0-roundtrip.bpmn:573: element process: Schemas validity error : Element '{http://www.omg.org/spec/BPMN/20100524/MODEL}process', attribute 'id': 'Process_ba16239e-181e-4b9f-bc5b-0bb2ee973450' is not a valid value of the atomic type 'xs:ID'.
   Signavio Process Editor 10.0.0/B.1.0-roundtrip.bpmn fails to validate
   

   Models after C.3.0 are missing which explains the lower number
4. Enterprise explorer produced invalid B.1.0: roundtrip and B.2.0: roundtrip too and additionally C.1.1: roundtrip

After that I stopped content that the system is in fact behaving.

[ghost]

With latest test result of ARIS the encoding issue is fixed, the schema validation counter is as expected (14/14).