CMLL normalized by number of query variables
| 30% Query, 0% Evidence | 30% Query, 10% Evidence | 30% Query, 20% Evidence | 30% Query, 40% Evidence | 30% Query, 50% Evidence | 10% Query, 30% Evidence | 20% Query, 30% Evidence | 30% Query, 30% Evidence | 40% Query, 30% Evidence | 50% Query, 30% Evidence |
| Data |
SPN CMLL | WM CMLL | DP CMLL | L1 CMLL | SPN CMLL | WM CMLL | DP CMLL | L1 CMLL | SPN CMLL | WM CMLL | DP CMLL | L1 CMLL | SPN CMLL | WM CMLL | DP CMLL | L1 CMLL | SPN CMLL | WM CMLL | DP CMLL | L1 CMLL | SPN CMLL | WM CMLL | DP CMLL | L1 CMLL | SPN CMLL | WM CMLL | DP CMLL | L1 CMLL | SPN CMLL | WM CMLL | DP CMLL | L1 CMLL | SPN CMLL | WM CMLL | DP CMLL | L1 CMLL | SPN CMLL | WM CMLL | DP CMLL | L1 CMLL |
| nltcs |
-0.48289 | -0.49560 | -0.52429 | -0.58923 |
-0.41594 | -0.41978 | -0.43327 | -0.45410 |
-0.35639 | -0.36043 | -0.36391 | -0.36338 |
-0.31413 | -0.31416 | -0.31320 | -0.31531 |
-0.30002 | -0.30104 | -0.29748 | -0.29795 |
-0.23166 | -0.23295 | -0.23397 | -0.23225 |
-0.37029 | -0.37208 | -0.37049 | -0.37176 |
-0.34769 | -0.34882 | -0.34734 | -0.35078 |
-0.38482 | -0.38686 | -0.38663 | -0.39005 |
-0.40454 | -0.40566 | -0.40722 | -0.40765
|
| msnbc |
-0.38159 | -0.38384 | -3.04022 | -0.37701 |
-0.38673 | -0.39079 | -2.23840 | -0.38365 |
-0.36741 | -0.37208 | -1.82149 | -0.36551 |
-0.36289 | -0.36674 | -1.35304 | -0.36277 |
-0.36493 | -0.36453 | -0.99913 | -0.37037 |
-0.23794 | -0.23515 | -1.04781 | -0.23734 |
-0.33226 | -0.33367 | -1.43659 | -0.33106 |
-0.37344 | -0.37501 | -1.65501 | -0.37407 |
-0.33089 | -0.33385 | -1.40032 | -0.33272 |
-0.35322 | -0.35642 | -1.53066 | -0.35290
|
| kdd |
-0.03033 | -0.02963 | -6.88711 | -4.47237 |
-0.03406 | -0.03464 | -3.74696 | -3.27954 |
-0.03005 | -0.03099 | -1.06902 | -2.04309 |
-0.03392 | -0.03388 | -0.10646 | -0.10506 |
-0.03183 | -0.03227 | -0.04240 | -0.03697 |
-0.03091 | -0.03028 | -0.18898 | -0.53116 |
-0.02787 | -0.02941 | -0.19845 | -0.45763 |
-0.03124 | -0.03134 | -0.18224 | -0.55894 |
-0.02931 | -0.02980 | -0.19556 | -0.53746 |
-0.03215 | -0.03285 | -0.17319 | -0.51020
|
| plants |
-0.43508 | -0.44200 | -2.21015 | -2.07440 |
-0.26328 | -0.27637 | -0.44382 | -0.72504 |
-0.20937 | -0.21611 | -0.25837 | -0.27525 |
-0.16850 | -0.16763 | -0.18922 | -0.18146 |
-0.15532 | -0.15679 | -0.17291 | -0.17003 |
-0.16316 | -0.16281 | -0.19448 | -0.18834 |
-0.16900 | -0.17656 | -0.20006 | -0.19876 |
-0.18960 | -0.19067 | -0.22379 | -0.21640 |
-0.19084 | -0.19269 | -0.22418 | -0.22567 |
-0.18524 | -0.18795 | -0.21693 | -0.21226
|
| baudio |
-0.48318 | -0.48150 | -0.48705 | -1.29149 |
-0.42466 | -0.42788 | -0.45102 | -0.46012 |
-0.40318 | -0.40365 | -0.42426 | -0.40952 |
-0.39190 | -0.38898 | -0.40545 | -0.38501 |
-0.39046 | -0.38655 | -0.39718 | -0.37832 |
-0.39720 | -0.39629 | -0.41421 | -0.39581 |
-0.39748 | -0.39580 | -0.41437 | -0.39599 |
-0.39970 | -0.39649 | -0.41342 | -0.39631 |
-0.39369 | -0.39322 | -0.40979 | -0.39388 |
-0.39899 | -0.39859 | -0.41576 | -0.39832
|
| jester |
-1.31657 | -1.27906 | -0.85827 | -1.87191 |
-0.80899 | -0.90610 | -0.69482 | -0.15217 |
-0.73435 | -0.75944 | -0.61933 | -0.13022 |
-0.72325 | -0.58990 | -0.52458 | -0.10509 |
-0.72521 | -0.53327 | -0.49447 | -0.09672 |
-0.72200 | -0.65693 | -0.56153 | -0.11494 |
-0.72205 | -0.65974 | -0.56265 | -0.11475 |
-0.72232 | -0.66230 | -0.56527 | -0.11566 |
-0.72241 | -0.65899 | -0.56362 | -0.11582 |
-0.72226 | -0.65831 | -0.56268 | -0.11529
|
| bnetflix |
-0.64571 | -0.64496 | -0.64659 | -0.66672 |
-0.60371 | -0.60649 | -0.62607 | -0.60451 |
-0.58723 | -0.59052 | -0.61629 | -0.58377 |
-0.57210 | -0.56622 | -0.59647 | -0.55051 |
-0.56058 | -0.55771 | -0.58716 | -0.53817 |
-0.57495 | -0.57756 | -0.60229 | -0.56363 |
-0.57633 | -0.57551 | -0.60430 | -0.56521 |
-0.57201 | -0.57112 | -0.60112 | -0.55825 |
-0.57492 | -0.57389 | -0.60246 | -0.56212 |
-0.57498 | -0.57245 | -0.60276 | -0.56154
|
| accidents |
-0.40393 | -0.40980 | -0.78369 | -4.91827 |
-0.36268 | -0.36846 | -0.65304 | -4.31739 |
-0.32438 | -0.32681 | -0.55221 | -3.42444 |
-0.27281 | -0.26190 | -0.40043 | -1.89813 |
-0.25414 | -0.23771 | -0.35356 | -1.19958 |
-0.31392 | -0.30901 | -0.50048 | -2.64131 |
-0.29623 | -0.29659 | -0.48024 | -2.62161 |
-0.30188 | -0.29860 | -0.47224 | -2.66820 |
-0.30170 | -0.29912 | -0.46772 | -2.67310 |
-0.29801 | -0.29205 | -0.46884 | -2.64418
|
| tretail |
-0.08211 | -0.08150 | -4.19211 | -2.99987 |
-0.08417 | -0.08392 | -0.56615 | -2.43826 |
-0.07906 | -0.07890 | -0.09621 | -1.83666 |
-0.07838 | -0.07763 | -0.07735 | -0.40206 |
-0.07698 | -0.07640 | -0.07639 | -0.10899 |
-0.07567 | -0.07509 | -0.07669 | -1.19562 |
-0.07820 | -0.07782 | -0.07797 | -1.16066 |
-0.07846 | -0.07787 | -0.08109 | -1.15064 |
-0.08059 | -0.08039 | -0.08018 | -1.20887 |
-0.08035 | -0.08026 | -0.08064 | -1.20575
|
| pumsb_star |
-0.46191 | -0.46620 | -1.38278 | -3.15674 |
-0.21807 | -0.23353 | -0.79155 | -1.47297 |
-0.17709 | -0.18430 | -0.55373 | -0.88527 |
-0.13373 | -0.12828 | -0.30725 | -0.34980 |
-0.12471 | -0.11358 | -0.25926 | -0.22551 |
-0.15760 | -0.16374 | -0.40988 | -0.58167 |
-0.15153 | -0.14804 | -0.40285 | -0.58215 |
-0.15544 | -0.15773 | -0.40414 | -0.56051 |
-0.15368 | -0.15413 | -0.41004 | -0.55151 |
-0.15488 | -0.15434 | -0.40516 | -0.57289
|
| dna |
-0.56616 | -0.55894 | -0.55819 | -0.56096 |
-0.55084 | -0.53715 | -0.55368 | -0.53774 |
-0.51076 | -0.51609 | -0.55144 | -0.51595 |
-0.46922 | -0.46990 | -0.54482 | -0.46818 |
-0.44952 | -0.45005 | -0.54389 | -0.44765 |
-0.49312 | -0.49533 | -0.54959 | -0.49459 |
-0.49507 | -0.49965 | -0.55507 | -0.49795 |
-0.49416 | -0.49708 | -0.55028 | -0.49588 |
-0.48886 | -0.49163 | -0.54689 | -0.49147 |
-0.49260 | -0.49486 | -0.54955 | -0.49448
|
| kosarek |
-0.06998 | -0.07012 | -0.07169 | -7.60305 |
-0.06068 | -0.06122 | -0.06658 | -6.13056 |
-0.05621 | -0.05641 | -0.06353 | -4.41242 |
-0.05524 | -0.05506 | -0.05997 | -0.08630 |
-0.05294 | -0.05378 | -0.05843 | -0.05624 |
-0.05534 | -0.05510 | -0.06329 | -1.61611 |
-0.05549 | -0.05605 | -0.06261 | -1.58083 |
-0.05735 | -0.05689 | -0.06376 | -1.67570 |
-0.05525 | -0.05551 | -0.06162 | -1.60876 |
-0.05524 | -0.05553 | -0.06077 | -1.60419
|
| msweb |
-0.03779 | -0.03737 | -0.03763 | -0.53502 |
-0.03632 | -0.03616 | -0.03645 | -0.33982 |
-0.03603 | -0.03545 | -0.03620 | -0.12717 |
-0.03320 | -0.03269 | -0.03339 | -0.03449 |
-0.03385 | -0.03312 | -0.03388 | -0.03359 |
-0.03695 | -0.03614 | -0.03706 | -0.05048 |
-0.03352 | -0.03286 | -0.03366 | -0.05088 |
-0.03697 | -0.03660 | -0.03734 | -0.05257 |
-0.03352 | -0.03321 | -0.03389 | -0.04318 |
-0.03516 | -0.03477 | -0.03533 | -0.04865
|
| book |
-0.08573 | -0.07812 | -0.07819 | -1.94326 |
-0.07084 | -0.07068 | -0.07940 | -1.31611 |
-0.06670 | -0.06606 | -0.07659 | -0.66640 |
-0.06600 | -0.06687 | -0.07654 | -0.07206 |
-0.06709 | -0.06863 | -0.07632 | -0.07107 |
-0.06833 | -0.06970 | -0.07856 | -0.09785 |
-0.06777 | -0.06772 | -0.07770 | -0.10277 |
-0.06738 | -0.06825 | -0.07817 | -0.10517 |
-0.06568 | -0.06618 | -0.07524 | -0.10656 |
-0.06657 | -0.06743 | -0.07691 | -0.09677
|
| tmovie |
-0.16812 | -0.17440 | -0.16737 | -2.81565 |
-0.11433 | -0.11584 | -0.16330 | -2.11255 |
-0.10467 | -0.10519 | -0.15241 | -0.56186 |
-0.10123 | -0.10175 | -0.14461 | -0.11616 |
-0.10002 | -0.10141 | -0.14045 | -0.10992 |
-0.10051 | -0.10121 | -0.14769 | -0.13891 |
-0.10148 | -0.10111 | -0.14938 | -0.13997 |
-0.10402 | -0.10343 | -0.14991 | -0.14240 |
-0.10231 | -0.10189 | -0.14785 | -0.13865 |
-0.10215 | -0.10179 | -0.14939 | -0.13760
|
| cwebkb |
-0.24100 | -0.21368 | -0.21349 | -2.21012 |
-0.19312 | -0.19283 | -0.21305 | -1.64899 |
-0.18960 | -0.18669 | -0.21316 | -0.96609 |
-0.18630 | -0.18374 | -0.21223 | -0.19374 |
-0.18410 | -0.18265 | -0.21008 | -0.18435 |
-0.18225 | -0.18043 | -0.20652 | -0.23085 |
-0.18758 | -0.18492 | -0.21275 | -0.23099 |
-0.18772 | -0.18353 | -0.21199 | -0.23023 |
-0.18777 | -0.18476 | -0.21339 | -0.23239 |
-0.18643 | -0.18346 | -0.21182 | -0.23354
|
| cr52 |
-0.12185 | -0.12053 | -0.11988 | -6.06216 |
-0.10008 | -0.10203 | -0.11955 | -4.77130 |
-0.09461 | -0.09617 | -0.11801 | -3.36517 |
-0.08992 | -0.09284 | -0.11722 | -0.28171 |
-0.09049 | -0.09424 | -0.11745 | -0.10222 |
-0.09200 | -0.09428 | -0.11857 | -1.89857 |
-0.09189 | -0.09350 | -0.11796 | -1.88441 |
-0.09180 | -0.09412 | -0.11741 | -1.88668 |
-0.09344 | -0.09524 | -0.11940 | -1.86591 |
-0.09219 | -0.09454 | -0.11851 | -1.88668
|
| c20ng |
-0.19175 | -0.18744 | -0.18679 | -5.36564 |
-0.17179 | -0.17030 | -0.18596 | -4.32237 |
-0.17030 | -0.16937 | -0.18757 | -3.20824 |
-0.16682 | -0.16886 | -0.18659 | -0.23779 |
-0.16563 | -0.16938 | -0.18570 | -0.17482 |
-0.16550 | -0.16623 | -0.18490 | -1.64460 |
-0.16968 | -0.16792 | -0.18701 | -1.69180 |
-0.16767 | -0.16775 | -0.18607 | -1.72434 |
-0.16643 | -0.16695 | -0.18589 | -1.68874 |
-0.16694 | -0.16717 | -0.18605 | -1.66341
|
| bbc |
-0.26896 | -0.26170 | -0.26121 | -0.26248 |
-0.24677 | -0.24896 | -0.26417 | -0.25905 |
-0.23373 | -0.23896 | -0.25816 | -0.24938 |
-0.23373 | -0.23871 | -0.26013 | -0.24720 |
-0.23077 | -0.23737 | -0.25663 | -0.24265 |
-0.23963 | -0.24298 | -0.26303 | -0.25346 |
-0.23499 | -0.23833 | -0.25956 | -0.24897 |
-0.23238 | -0.23771 | -0.25929 | -0.24701 |
-0.23442 | -0.23896 | -0.25994 | -0.24847 |
-0.23512 | -0.23884 | -0.25948 | -0.24757
|
| ad |
-0.04296 | -0.04076 | -0.04222 | -0.37202 |
-0.01854 | -0.02007 | -0.04075 | -0.19348 |
-0.01219 | -0.01328 | -0.04136 | -0.11090 |
-0.00921 | -0.00958 | -0.03787 | -0.03112 |
-0.00819 | -0.00885 | -0.03667 | -0.01645 |
-0.01073 | -0.01182 | -0.04232 | -0.06560 |
-0.01016 | -0.01085 | -0.03871 | -0.05764 |
-0.01056 | -0.01083 | -0.03945 | -0.06203 |
-0.01041 | -0.01042 | -0.03994 | -0.06509 |
-0.01061 | -0.01127 | -0.03912 | -0.06482
|
Number of datasets where SPN has higher CMLL
| WM | DP | L1 |
| 30% Query, 0% Evidence | 7 | 10 | 17
|
| 30% Query, 10% Evidence | 14 | 19 | 17
|
| 30% Query, 20% Evidence | 15 | 19 | 17
|
| 30% Query, 40% Evidence | 9 | 17 | 15
|
| 30% Query, 50% Evidence | 11 | 17 | 14
|
| 10% Query, 30% Evidence | 10 | 19 | 16
|
| 20% Query, 30% Evidence | 10 | 18 | 16
|
| 30% Query, 30% Evidence | 11 | 18 | 17
|
| 40% Query, 30% Evidence | 12 | 18 | 18
|
| 50% Query, 30% Evidence | 11 | 19 | 16
|
| Avg. SPN wins | 11 | 17.4 | 16.3 |