Release 5.4: 🚚 Variable SERVICE targets, major performance improvements, and stricter spec compliance

In this minor release, we focused on a variety of improvements, such as better performance, stricter spec compliance, and support for variable SERVICE targets.

🚀 Improved cost model for better performance

Besides flexibility, performance is of high importance of Comunica. That is why we keep a close eye to it through continuous performance tracking and large scale benchmarks.

Across the WatDiv and BSBM benchmarks, we observed that the cost model used for query optimization was not always accurate. This could lead to suboptimal query plans being chosen, which in turn could lead to worse performance. In this release, we improved the cost model to better reflect the actual costs of query execution. The overall runtime of WatDiv and BSBM became more than 2x faster with these changes. Especially (very) slow queries have become significantly faster.

Concretely:

• Shared variables across subqueries are taken into account more precisely.
• Non-bind-joins with 3+ entries are estimated more accurately.
• The scale of costs within the bind-join is more accurate.
• Cardinalities in RDF/JS sources can be cached.
• Join cardinalities use distinct values when available.

Besides this, we also introduced tweaks that make ORDER BY faster in general, and introduced an optimization when ORDER BY is used in combination with LIMIT and OFFSET, which can lead to significant performance improvements for certain queries.

Below, you can see the evolution of performance improvements going back to May 2024, with significant reductions in the last few commits.

🚚 Variable SERVICE targets

SPARQL's SERVICE operator allows users to define a target source for a subquery. While we already had support for this operator, it was not possible to use variables as the target source. This is not strictly required by the SPARQL spec, but we received a number of requests for this feature, and it can be useful in certain scenarios. In this release, we added support for variable SERVICE targets, which enables queries such as the following to be written:

PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX foaf: <http://xmlns.com/foaf/0.1/>

SELECT * WHERE {
  ?person a foaf:Person;
          rdfs:seeAlso ?sa.
  SERVICE ?sa {
    ?person foaf:knows ?other.
  }
}

🤓 Stricter spec compliance

While Comunica already passed all SPARQL 1.1 specification tests, there were still a number of edge-cases that were not handled correctly. In this release, we fixed many of these issues, such as:

• Better handling of edge-cases in langMatches
• Stricter handling of ORDER BY
• Optional support for ordering of invalid and non-literals.
• Better handling of OPTIONAL, FILTER, and MINUS

🔺 Increased robustness for our SPARQL endpoint

When running Comunica as a SPARQL endpoint (e.g. using comunica-sparql-http), we now have support for the new HTTP QUERY method. This is especially useful for queries that are too large to fit in a URL, as it allows clients to send the query in the request body instead of the URL, while still being cacheable. Also the reverse, when Comunica queries over an endpoint that supports the HTTP QUERY method (through the Accept-Query header), is now supported as well.

Furthermore, the endpoint has been made more robust against certain types of invalid requests, and will now return a proper 400 Bad Request response instead of a worker crashing. Also some types of update requests don't hang anymore.

🤝 Contributors

This release has been made possible thanks to the help of the following contributors (in no particular order):

• voidten
• Jonni Hanski
• redreye
• Noah Van Steenbrugge
• Jim Balhoff
• Jitse De Smet
• Ruben Taelman

If you would like to contribute yourself, be sure to have a look at our contribution guide. We even have some new bounties that allow you to get paid for your contribution!

Full changelog

If you want to learn more about the other changes in Comunica 5.4, check out the full changelog.