Comunica Iterator Utils

Utilities related to AsyncIterator.

This module is part of the Comunica framework, and should only be used by developers that want to build their own query engine.

Click here if you just want to query with Comunica.

Install

$ yarn add @comunica/utils-iterator

Exposed utilities

• ChunkedIterator: Splits an iterator into chunks based on a given block size.
• ClosableIterator: An AsyncIterator with a callback for when this iterator is closed in any way. In contrast to ClosableTransformIterator, this does not add the overhead of a TransformIterator.
• ClosableTransformIterator: A TransformIterator with a callback for when this iterator is closed in any way.
• instrumentIterator: Profile an iterator by monkey-patching its _read and read methods.

Changelog

All notable changes to this project will be documented in this file.

v4.0.1 - 2024-10-15

BREAKING CHANGES

The following breaking changes will only impact users of Node.js < 18 and users that develop custom components or engines with Comunica.

• Remove backwards-compat for Node.js < 18
  • Remove cross-fetch and abort-controller polyfill as unnecessary
  • Update fetch-sparql-endpoint to version 5
  • Remove Node 14 backwards-compat in stats writer
• API changes
  • Refactor Actor.test to not rely on Promise rejections
    • All Actor.test methods must be changed to use passTest or failTest instead of promise rejections, for better performance.
  • Use variable.canBeUndef in metadata instead of canContainUndefs
  • Remove Actor.initialize and .deinitalize
  • Rename noCache CLI argument to invalidateCache
  • Remove hash collisions flag from hash buses
  • Return 32-bit numbers in hash bus based on specific variables
  • Allow inserts and deletes to execute in a single request
• Renamed or changed actors
  • Split expression functions into separate actors on dedicated bus
  • Split aggregators into separate actors on dedicated bus
  • Refactor HTTP fetch and retry logic
    • Retries of failed requests has been improved
    • The User-Agent header has been improved
    • A better approach to detect if Comunica is in a browser environment, compatible with Node.js, Deno, and Bun.
  • Remove SHA1 hash quads actor in favor of Murmur3
  • Remove SHA1 hash actor in favor of Murmur3
  • Abstract actor-rdf-join-minus-hash-undef into actor-rdf-join-minus-hash
• Move utilities
  • Add utils-query-operation and utils-iterator
    • These new packages contain helper functions that previously existed in bus-query-operation and bus-rdf-join.
  • Rename util packages to 'util-' scope
    • data-factory -> utils-data-factory
    • bindings-factory -> utils-bindings-factory
    • jest -> utils-jest
    • metadata -> utils-metadata
  • Move ActorRdfJoin.hashNonClashing to bindingsToCompactString
  • Delegate all join hashing to hash bus
  • Move bindings index from optional join to utils package

Added

• Add default Accept-Encoding header with Brotli
• Add undef support to inner hash join actor
• Forward side data from test to run in actors
• Allow plugging in custom DataFactory through context, Closes #358

Changed

• Remove unneeded false default values on CLI boolean args
• Improve fragment selector shape handling
  • Make doesShapeAcceptOperation recurse into operations
  • Add dedicated wildcard operation type to FragmentSelectorShape
• Improve performance
  • Bump asyncjoin with hash join performance improvement
  • Only consider overlapping vars when testing undef in join actors
• Improve type inference in IActionContext, Closes #933
• Emit planning time from stats writer
• Make stats writer measure time from actual query start
• Improve error reporting when mediators fail

Fixed

• Fix union not setting canContainUndefs to true if child vars differ

v3.3.0 - 2024-10-08

Added

• Add statistics tracking

Changed

• Optimize Bindings merge logic

Fixed

• Fix source attribution for QPF and SPARQL endpoints (#1420)
• Fix internal cardinalities being wrong for SPARQL endpoints with VoID
• Define inner join actor orders for NaN cardinalities

v3.2.3 - 2024-08-22

Fixed

• Fix filter pushdown into patterns losing variable bindings

v3.2.2 - 2024-08-19

Changed

• Optimizations:
  • Splice federated left join with empty right-hand input
  • Add ClosableIterator that bypasses TransformIterator
  • Only enable filter pushdown if criteria are met
  • Push down simple equality filters into triple patterns
  • Support pushing filters into left joins
  • Split conjunctive filters before pushdown
  • Merge nested filters into conjunctive after pushdown

Fixed

• Fix empty optionals in federation producing empty results for full query
• Don't push down FILTER (NOT) EXISTS

v3.2.1 - 2024-08-07

Added

• Add support for deduplication of results in construct queries

Changed

• Remove projection variables missing warning
• Migrate parse-link-header to http-link-header
• Migrate stream-to-string and streamify-string
• Performance improvements:
  • Tweak default BindJoin.minMaxCardinalityRatio value for WatDiv-C1
  • Use Bind Join only if smallest much smaller than largest
  • Assign smallest stream as build side in Hash Join, #884
  • Force NLJ when join entries have no overlapping variables

Fixed

• Fix optimizer incorrectly considering some optionals empty
• Fix SLICE actor not working with ASK queries, Closes #1374
• Make projection variables missing error a warning instead
• Fix some unions in federated queries being pruned incorrectly, Closes #1397
• Avoid single-entry joins in join-connected optimizer, #1397
• Consider unions when determining if operation can produce undefs, #1406
• Fix undefined values from SPARQL endpoints being produced
• Fix unnecessary join being created when binding over filter
• Fix initialBindings failing when done over projected variables

v3.2.0 - 2024-07-05

Added

• Performance improvements:
  • Add optional hash join actor
• Add actors for bindings source attribution
• Add OPT+ operator

Changed

• Performance improvements:
  • Lower default ioWeight to tweak overall performance
  • Make optional hash and bind join only work with common variables
• Debugging and performance tracking:
  • Omit undefined join coefficients in debug logger
  • Emit timings and actual cardinalities in physical explain output
  • Make physical query plan explain output compact and human-readable
  • Compact Bind-Join operations in physical query explain output
  • Add BJ bindOperation to physical query explain output
  • Emit pattern source in physical query explain output

Fixed

• Fix undef sanity check for SPARQL endpoints sometimes going rogue
• Fix CONSTRUCT FROM not being sent to endpoints, Closes #1389
• Fix empty media type not falling back to extension handling
• Fixed the order of count operations influencing the result
• Fix minus-hash using incorrect hash function

v3.1.2 - 2024-05-30

Fixed

• Fix next page requests not being logged anymore
• Fix performance regression of TPF queries

Changed

• Improve performance:
  • Use AsyncIterator.map and .filter instead of expensive .transform
  • Improve performance of hashing during hash joins
  • Optimize usage of bindingsStream transform in PathNps actor
• Use event-emitter-promisify package
• Cleanup stream operations (#1349)
• Maintain backpressure in result serializers
• Maintain backpressure on internal actors
• Don't log if no sources are pruned
• Use cancel in wayback actor when available, Closes #1351

v3.1.1 - 2024-05-11

Fixed

• Fix queries over paged collections always being empty

v3.1.0 - 2024-05-11

Added

• Add @comunica/query-sparql-rdfjs-lite that is optimized for bundle size

Changed

• Migrate from readable-web-to-node-stream to readable-from-web
• Move validateHttpResponse helper to avoid interdependencies

Fixed

• Avoid race conditions during DELETE/INSERT operations
• Catch 404 of non-existing resource when fetching the selector shape (#1324)
• Always add preprocessed context to unidentified query sources (#1337)

v3.0.3 - 2024-04-12

Changed

• Add generic type to LinkQueueWrapper (#1322)

Fixed

• Fix broken LIMITs on CONSTRUCT queries on SPARQL endpoints, Closes #1319

v3.0.2 - 2024-04-10

Fixed

• Lower q-value of application/json in fetch requests

v3.0.1 - 2024-03-19

BREAKING CHANGES

• Rename sparqlee-specific package names
  • @comunica/actor-query-operation-filter-sparqlee -> @comunica/actor-query-operation-filter
  • @comunica/actor-query-operation-orderby-sparqlee -> @comunica/actor-query-operation-orderby
• Generalize sources to accept all query operations
  • The rdf-resolve-quad-pattern bus has been removed in favor of the new query-source-identify bus. This bus enables identification of IQuerySource's which can accept any kind of query operation, not just quad patterns like before.
  • brTPF support is added.
  • Bound joins are supported: pushing bindings into sources (FedX)
  • Several context-preprocess actors are added due to this, which offer:
    • Grouping operations into sources during federated querying (FedX)
    • Pushing down filters into sources
    • Skolemization of blank nodes
• Move sequential query process steps to separate bus and actor
• Move context shortcut handling to preprocess actor

Added

• Allow Bindings to have contexts
• Add timeout for sparql endpoint count queries
• Process source contexts to support shortcuts
• Add option to disable caching (#1297)
• Allow sources to be annotated with traverse flag

Fixed

• Fix GROUP_CONCAT losing languages if equal

Changed

• Update to asynciterator 3.9.0 to make results async iterable
• Ensure QueryEngine instances are fully independent
• Update to Components.js v6

v2.10.2 - 2024-01-09

Fixed

• Migrate to @smessie/readable-web-to-node-stream

v2.10.1 - 2023-11-29

Fixed

• Fix updates with body streams failing in Node >= 18
• Generalize fromBindings in BindingsFactory

v2.10.0 - 2023-10-27

Added

• Convert raw source contexts to ActionContexts
• Support relative and absolute paths for COMUNICA_CONFIG

Changed

• Log expression error messages
• Print log levels in different colors
• Update readable-stream, yargs, lerna, typescript and eslint (#1271)

Fixed

• Fix wrong cardinality for zero-or-one paths with one variable
• Support building with esModuleInterop TS flag
• Skip calling endsWith on mediaType when mediaType is undefined

v2.9.0 - 2023-09-07

Changed

• Early close/destroy in hypermedia iterator when aggregatedStore is unused
• Update dependency uuid to v9 (#1247)
• Migrate sparqlee into Comunica

Fixed

• Enable SPARQL-star for update queries to endpoints

v2.8.3 - 2023-08-21

Changed

• Throw when doing BIND on a pre-bound variable, Closes #1229
• Include final SPARQL endpoint URL in debug logger

Fixed

• Fix hypermedia errors being voided, Closes #1241
• Fix some SPARQL endpoint errors being voided
• Fix missing end event for empty queries with limit
• Fix process halt when re-executing query over erroring source

v2.8.2 - 2023-08-10

Fixed

• Fix aggregated store not containing first source after query re-exec

v2.8.1 - 2023-07-14

Fixed

• Allow multi join actors to handle undefs, Closes #1231
• Fix process halting on many TPF sources

v2.8.0 - 2023-07-04

Added

• Add support for RDF-star and SPARQL-star
  • Support quoted triples in result serializers
  • Add support for processing quoted triples

Changed

• Update to rdf-store-stream v2
• Update dependency lru-cache to v10
• Update to TypeScript 5.x

v2.7.1 - 2023-05-31

Changed

• Update dependency htmlparser2 to v9 (#1211)
• Simplify stringSource actor implementation

Fixed

• Fix existence hook not provided error with OPTIONAL, Closes #1029
• Fix stringSource syntax errors not being caught, Closes #1192
• Fix bind join illegally being used over FILTERs
• Fix NaN selectivity for non-pattern operations

v2.7.0 - 2023-05-24

Added

• Support durations, dates, and times in filters
• Add support for ZeroOrOne path expressions with 2 variables

Changed

• Improving Web support
  • Use process shim (#1186)
• Improve performance with SPARQL endpoints
  • Improve metadata and lazyness in SPARQL hypermedia actor
  • Fix SPARQL endpoint operator always having canContainUndefs: true
  • Fix keepalive not being true in fetch-only environments
  • Cache count queries over SPARQL endpoints
• Enable incremental metadata updates
• Update dependency canonicalize to v2
• Make Bindings#merge and mergeWith accept more generic RDF.Bindings

Fixed

• Fix blank node scoping in CONSTRUCT queries, Closes #1190
• Add missing baseIRI for shaclc
• Destroy unused iterators

v2.6.10 - 2023-03-10

Fixed

• Add missing shaclc dependency in rdfjs engine

v2.6.9 - 2023-03-08

Changed

• Update to Node.js typings v18

Fixed

• Bump rdf-streaming-store with another fix for stream.push() after EOF
• Use globalThis instead of global
• Fix OPTIONAL failing to emit FILTERed results

v2.6.8 - 2023-03-06

Changed

• Remove deprecated calls to lru-cache

Fixed

• Bump rdf-streaming-store with fix for stream.push() after EOF
• Fix aggregated store not ending when linked iterator is destroyed
• Scope aggregated stores in the query context

v2.6.7 - 2023-02-15

Added

• Allow minimum sources check on CLI to be disabled
• Allow generic context type in QueryEngine to omit fields

v2.6.6 - 2023-02-08

Fixed

• Prefer piping in serializers to maintain backpressure, #759

Changed

• Update dependency lru-cache to v7
• Don't lookup default graph when none available in QPF

v2.6.5 - 2023-02-07

Fixed

• Fix group operator only returning stream after completion
• Fix incorrect caching of QPF requests on default graph

v2.6.4 - 2023-02-02

Fixed

• Fix aggregated hypermedia store not closing with LIMIT

v2.6.3 - 2023-02-01

Fixed

• Fix aggregated hypermedia store halting if initial stream is destroyed

v2.6.2 - 2023-01-31

Fixed

• Fix shaclc parser not being able to pipe

v2.6.1 - 2023-01-30

Fixed

• Fix aggregated hypermedia store halting if initial stream not fully consumed
• Cache parsed sources in string source actor

v2.6.0 - 2023-01-26

Added

• Add SHACLC parser and serializer
• Allow hypermedia actor to aggregate link queue in an index

Changed

• Use sparqlee's new type system

Fixed

• Fix invalid baseIRI when querying local files
• Fix errors in Node 19

v2.5.2 - 2022-11-18

Fixed

• Fix broken bindingsStreamToGraphQL export
• Fix incorrect bindingsStreamToGraphQl return type

v2.5.1 - 2022-11-16

Fixed

• Add missing dependencies in file and rdfjs engines
• Rename wayback actor in config, Closes #1086
• Add localizeBlankNodes to IQueryContextCommon
• Fix incorrect files entry in data-factory
• Add recoverBrokenLinks to IQueryContextCommon

v2.5.0 - 2022-11-09

Added

• Enable engines to override context shortcuts and typings
• Support string-based data sources, Closes #639
• Add support for HTTP retries
• Add wayback HTTP actor

Changed

• Add USER to Dockerfile
• Make QueryOperationAsk close the BindingsStream earlier
• Expose localizeBlankNodes on the CLI

Fixed

• Bump to asynciterator 3.8 with ESM packaging fix, Closes #1097
• Include source map files in packed files
• Fix invalid blank node skolemization in DESCRIBE queries

v2.4.3 - 2022-09-06

Fixed

• Add workaround for INSERT queries on Solid pods
• Add polyfill for Buffer in webpack

v2.4.2 - 2022-08-26

Fixed

• Fix typings issue in ActorDereferenceParse#handleDereferenceStreamErrors

v2.4.1 - 2022-08-24

Fixed

• Make global process override conditional

v2.4.0 - 2022-08-24

Added

• Add logging on timeout handlers in HTTP service
• Emit metadata in SPARQL/JSON serializer
• Enable context overriding when running SPARQL endpoint

Changed

• Move away from Node.js built-ins to avoid polyfilling in tools such as Webpack
  • Replace setImmediate by setTimeout
  • Use performance.now instead of process.hrtime
  • Avoid using htmlparser2 stream feature
  • Remove xml dependencies
  • Remove dependency on web-streams-node
• Performance improvements:
  • Update to AsyncIterator 3.7.0
  • Rewrite */+ property paths with proper backpressure handling

Fixed

• Fix propagation of iterator destroying in various places
• Fix blocking operators delaying timeout start in HTTP service
• Fix halting when error occurs in extend operation
• Avoid applying @babel/polyfill on the output code
• Close connections on uncaught exceptions in HTTP service workers
• Kill HTTP service master process if worker was forcefully killed

v2.3.0 - 2022-06-29

Added

• Add httpTimeout option to timeout on HTTP requests
• Add httpTimeoutOnBody option
• Add options to run using union default graph semantics
• Add freshWorker option to HTTP service

Changed

• Improve warning messages in extend actor
• Properly sort named nodes, blank nodes and unbounded values

Fixed

• Fix race conditions when timing out in HTTP server
• Fix responses not always being sent after HTTP server timeout
• Fix closing of streams not always propagating
• Fix N3.js parser not receiving parsing media type
• Fix physical query plan crash when multi-smallest inner join is used

v2.2.1 - 2022-04-13

Fixed

• Fix request with body in browser not working

v2.2.0 - 2022-04-12

Added

• Add getSafe to ActionContext and bus convenience interfaces
• Export deskolimisation functions (#974)

Changed

• Require minimum @rdfjs/types version (#948)
• Update to arrayify-stream 2
• Update to immutable.js 4
• Add missing deps in bus-dereference

Fixed

• Fix invalid JSON in a config file
• Fix hash joins not working on queries with limits
• Fix crash for requests with string body, Closes #969

v2.1.0 - 2022-03-09

Added

• Allow links to be followed in parallel

v2.0.6 - 2022-03-07

Fixed

• Remove original query string from context for initialBindings, Closes #941

v2.0.5 - 2022-03-04

Fixed

• Fix unhandled parse errors, Closes #872

v2.0.4 - 2022-03-04

Fixed

• Disable fetch keepalive when sending a body

v2.0.3 - 2022-03-02

Fixed

• Bump deps to avoid old transitive Comunica deps

v2.0.2 - 2022-03-02

Fixed

• Fix extension-based media type identification not working
• Fix invalid Dockerfile

v2.0.1 - 2022-03-02

BREAKING CHANGES

• Set minimum Node version to 14
  • Update tsconfig to es2020
• Public API
  • Improve names of query result interfaces
  • Implement new RDF/JS Bindings and BindingsFactory
  • Rename update results to void results
  • Implement RDF.Queryable in IQueryEngine
  • Make query context type-safe
  • Represent query formats as RDF.QueryFormat
  • Implement RDF.SparqlQueryable in IQueryEngine
• Internal API
  • Make metadata in query operations mandatory and strictly typed
  • Add Mediate utility type, Closes #902
  • Make ActionContext type-safe
  • Make context object required in actions
  • Use Headers rather than Record<string, string>
  • Rename IQueryableResult to IQueryOperationResult
  • Make IMediatorTypeJoinCoefficients implement RDF.QueryOperationCost
  • Migrate metadata interface to RDF/JS metadata types
  • Move variables field into metadata
  • Make boolean and void results lazy
• Remove deprecated features
  • Remove deprecated string variant in links bus output
  • Make links queue mediator in hypermedia resolver mandatory
  • Remove backwards-compat for source and hypermedia tagged sources
  • Remove deprecated exports
  • Remove unused init actors
  • Remove deprecated packages
  • Remove inefficient query-based metadata extractors
  • Remove utils-datasource package
  • Remove bindings and quads methods in favor of toArray on AsyncIterator
• Restructure packages
  • Move index files to lib directory
  • Rename actor-rdf-dereference-http-parse to actor-rdf-dereference-http
  • Move peerDependencies to dependencies
  • Remove unnecessary entries from package.json files
  • Rename bus-sparql-serialize to bus-query-result-serialize
  • Rename bus-sparql-parse to bus-query-parse
  • Rename actor-http-node-fetch to actor-http-fetch
  • Decouple configs and query engines into separate packages
  • Add generic dereference bus
  • Move sorting of join entries to dedicated bus
  • Remove args_ prefixes in config files when possible
• Refactor join operations:
  • Add logical join type field to join action
  • Add actor to cluster connected join entries
  • Consider multiple coefficients during join mediation
  • Add optimize actor to convert BGPs to joins
  • Add cardinality metadata to all property path actors
  • Rename totalItems metadata to cardinality
  • Remove BGP actors in favor of new join actors
  • Add BGP actor that delegates to join bus
  • Add Bind-Join actor
  • Add bus for providing join selectivities
  • Add variable counting join cardinality heuristic
  • Add minus query operation actor that delegates to join bus
  • Add left join query operation actor that delegates to join bus
  • Add hash-based minus join actor that does not handle undefs
  • Add hash-based minus join actor that handles undefs
  • Add bind-based optional join actor
  • Add nested loop-based optional join actor
  • Change BGP-specific context entries to more general join entries
  • Include original operations into join input interface
  • Migrate to multi-join-based datastructures in sparqlalgebrajs
  • Rename plain join actors to inner join actors
• Other
  • Deskolemise blank nodes before updates, Closes #911

Added

• Public API
  • Allow query plan to be printed
  • Add bindings-factory package
  • Allow full stack trace to be printed on CLI via --showStackTrace
  • Add dedicated explain method on IQueryEngine
• Internal API
  • Expose convenience types in bus packages for simpler actors impls
  • Add hash provider bus
  • Allow packager to output list of dependencies for a config
  • An optional parameter to MediatorCombinePipeline to filter erroring actors
  • Add option to enable ordering strategy in MediatorCombinePipeline
  • Add setSafe method to IActionContext

Changed

• Automatically generate components/ directories:
  • Remove components directory from git
  • Generate components in bulk
• Move index files to lib directory
• Update to sparqlalgebrajs 4
• Mark parsed quad stream as Readable
• Send original SPARQL query string to endpoints
• Update to Webpack 5
• Add sideEffects: false to package.json files for treeshaking
• Prefer joins with early results for queries with limit

v1.22.3 - 2021-10-19

Fixed

• Fix CORS issues when querying in Firefox
• Fix media type overriding not working for HTTP requests
• Fix source type detection not always working in SERVICE

v1.22.2 - 2021-09-30

Changed

• Include HTTP response in update error messages
• Allow patchSparqlUpdate destinations to not exist when forced
• Extract ms-author-via header for SPARQL updates, to support older Solid servers
• Pass optional headers in dereference errors

Fixed

• Fix fetch actor ignoring request bodies in Node.js
• Add temporary workaround for loss of Headers on Solid auth requests

v1.22.1 - 2021-09-22

Fixed

• Fix rare stream interruption before end

v1.22.0 - 2021-08-30

Added

• Add support for SPARQL extension functions
• Allow custom fetch function to be provided via context
• Support AbortControllers in native http actor
• Report FILTER errors as warnings in the logger, Closes #767
• Add SPARQL endpoint-based hypermedia update actor
• Allow resource creation via HTTP PUT:
  • Add RDF update hypermedia actor for LDP PUT destinations
  • Configure RDF update hypermedia actor for LDP PUT destinations
  • Allow RDF dereference actors to return on non-200 responses
  • Add metadata extractor for HTTP Accept-Put header
  • Add metadata extractor for HTTP Allow header

Changed

• Refactor CLI arguments handling using yargs
• Optimize node-fetch HTTP actor and set as default
• Include response body in dereference errors
• Bump fetch-sparql-endpoint with improved error reporting for SPARQL endpoints
• Update dependency htmlparser2 to v7 (#851)
• Migrate to @rdfjs/types
• Bump to sparqlalgebrajs 3
• Notify downstream clients on worker timeout
• Bump fetch-sparql-endpoint with AbortControllers
• Make HTTP accept headers deterministic
• Add URL-based heuristic to determine SPARQL endpoints to top-level actor

Fixed

• Update to AsyncIterator 3.2.0 to fix memory issue
• Fix http proxy not working with immutable node-fetch response
• Fix invalid hash implementation for hash-join
• Fix node-fetch actor not supporting body.cancel
• Do not reset sparql.js blank-node counter (Closes #836)
• Fix SPARQL-based update actors not being able to handle quads

v1.21.3 - 2021-06-18

Fixed

• Fix POST requests with body not working on browsers

v1.21.2 - 2021-06-14

Changed

• Interact with SPARQL endpoints via HTTP POST, Closes #814
• Update dependency @types/node to v14
• Update jsonld-streaming-parser to 2.3.2

v1.21.1 - 2021-04-27

Republish due to npm publishing failure of 1.21.0.

v1.21.0 - 2021-04-27

Added

• Enable hypermedia-driven updates
  • Pass headers to metadata extraction actors
  • Add metadata extractor for detecting SPARQL Update Patch servers
  • Configure hypermedia-based update actors
  • Add actor for handling servers supporting SPARQL Update patches
  • Add update actor for delegating to hypermedia bus
  • Add bus for hypermedia-based update actors
• Support HTTP requests with bodies
• Allow context modification in optimize-query-operation bus

Changed

• Move IQuadDestination to separate file
• Allow multiple graphs to be created/deleted in IQuadDestination
• Use --to option instead of -d to mark destination on CLI
• Allow any parser options to be passed to actor-rdf-parse-jsonld

Fixed

• Emit non-zero exit code on CLI errors, Closes #805
• Remove unnecessary dependencies on utils-datasource to mute Yarn warnings
• Fix types dependencies not being actual dependencies
• Fix unable to federate over multiple GraphDB instances

v1.20.0 - 2021-03-30

Added

• Add support for SPARQL Update queries
  • Add update actors to all init engines
  • Support update queries in SPARQL endpoint
  • Support update results in simple serializer
  • Add readOnly context entry that makes update actors fail
  • Allow single source in context to be inherited as destination
  • Add add, copy, and move update operation actors
  • Add create update operation
  • Add clear and drop update operations
  • Configure load update actor
  • Add load update query operation actor
  • Add fallback rdf-dereference actor
  • Run SPARQL Update test suite
  • Enable passing destination via the CLI
  • Configure INSERT and DELETE actors
  • Add composite update operation actor
  • Add INSERT/DELETE operation actor
  • Add RDF update actor that handles RDF/JS stores
  • Add bus for handling quad updates
  • Add update query operation output type
• Enable worker threads in SPARQL endpoints
• Delegate link queue creation to the new link queue bus

Changed

• Move public query engine interfaces into @comunica/types
• Move all context keys to '@comunica/context-entries'
• Keep track of parent link when pushing into link queue
• Save original query in context

Fixed

• Enable blank node correlation across results, Closes #795
• Fix FROM/FROM NAMED not always restricting views
• Bump asyncjoin with close on undefined fix, Closes #791
• Fix process hanging if a source parsing action rejects, Closes #786
• Fix hypermedia quad pattern resolver not skipping links to itself

v1.19.2 - 2021-02-02

Changed

• Remove disabling of stacktraces when in production

Fixed

• Fix unhandled promise rejection on server errors

v1.19.1 - 2021-01-22

Fixed

• Fix logger not always producing output, Closes #774

v1.19.0 - 2021-01-15

Changed

• Update to Components.js 4
• Update dependency htmlparser2 to v6

v1.18.1 - 2020-12-01

Fixed

• Fix HTTP service not handling conneg correctly with wildcards
• Remove unsafe usages of process
• Fix minor (non-breaking) issues with configs

v1.18.0 - 2020-11-02

Fixed

• Fix eager accept header boundary bug, Closes #534
• Fix incorrect return type for bindings(), Closes #742
• Add missing query result interface exports

Added

• Add Microdata to RDF parser
• Add actor whitelist to LoggerPretty

Changed

• Use fixed hashing functions
• Update sparqlee to achieve smaller bundle sizes
• Bump sparqlee with improved XPath and Unicode support
• Allow quad streams on followed links to be transformed
• Allow hypermedia links to define custom context entries
• Allow JSON-LD strict mode to be enabled via context
• Allow custom JSON-LD document loader to be passed via context

v1.17.0 - 2020-09-25

Added

• Include request headers in logs, Closes #703
• Include user agent in node-fetch HTTP actor

Changed

• Update to asynciterator 3.0.3 to fix hanging queries
• Make metadata retrieval more robust, and fix errors where metadata is emitted before end event
• Make logger data argument lazy
• Tweak priorities to let endpoint default to application/json, Closes #662
• Increase default Turtle priority to 0.6, higher than RDF/XML
• Update to @types/rdf-js 4
• Migrate to rdf-data-factory

Fixed

• Update to JSON-LD context parser with schema.org fix
• Fix redirect not including query parameter

v1.16.2 - 2020-08-24

Fixed

• Fix native HTTP actor broken in browser

v1.16.1 - 2020-08-24

Fixed

• Fix incorrect JSON-LD error code on target script error

v1.16.0 - 2020-08-24

Added

• Implement missing property path cases
• Add HTTP basic authentication support, Closes #613
• Add TSV SPARQL results serializer, #220
• Add CSV SPARQL results serializer, #220

Fixed

• Update sparql parser to fix all syntax spec test failures, #287
• Fix network request without headers, Closes #700 (#701)
• Correctly handle aggregates without group
• Fix subquery aggregates halting the process
• Fix variable graph also matching with default graph
• Fix hypermedia actor failing on * property path queries
• Fix property path queries failing with named graphs
• Fix mainModulePath overriding not being possible in dynamic exec
• Fix process hanging after HTTP error
• Fix HTTP error events being emitted twice
• Fix targeted HTML script tags not throwing error if of unknown type
• Fix missing newline after --listformats

Changed

• Configure nestedloop join actor for handling streams with undefs
• Annotate bindingsStreams with canContainUndefs flag

v1.15.0 - 2020-08-13

Added

• Add convenience methods for getting query results as arrays
• Add option to include credentials in browsers
• Enable min/max over non-numerical literal, Closes #673

Changed

• Improve command signature of comunica-sparql-http
• Remove lodash dependencies, Closes #402

Fixed

• Fix error when parsing webpage with JSON script tag, Closes #707
• Update to asynciterator 3.0.1 to undo webpack hack
• Add missing newline after each pretty log line
• Fix incorrect hash function in symmetric hash join, Closes #608
• Bump to sparqlee 1.4.2 with strtswith and endswith fix

v1.14.0 - 2020-07-24

Added

• Enable cardinality estimates in RDFJS sources
• Redirect default HTTP index to /sparql, Closes #663
• Simplify how RDFJS sources can be passed

Changed

• Migrate to asynciterator to 3 to improve performance in browsers
• Update dependency lru-cache to v6
• Move relevant @types to dependencies, Closes #294
• Replace json-stable-stringify dependency with canonicalize, Closes #590

Fixed

• Fix floating promises
• Fix test failures on Node 10
• Use latest fetch API with cross-fetch to fix fetch-related compatibility issues
• Fix orderby not taking numerical types into account
• Fix invalid RDFJS actor IRI
• Fix newEngine in SPARQL init file exporting browser build, Closes #648
• Fix orderby not properly handing multiple comparators, Closes #598

v1.13.1 - 2020-06-11

Fixed

• Fix crash when using CONSTRUCT queries
• Fix unasigned variables in projection result

v1.13.0 - 2020-06-03

Fixed

• Modify JSON-LD document loader to consider redirects
• Fix table serializer producing inconsistent column orders, Closes #643
• Update sparqlalgebra to fix invalid COUNT queries to endpoints

Changed

• Enable stricter TypeScript compilation, Closes #628

v1.12.1 - 2020-04-27

Fixed

• Fix EXISTS in BIND failing, Closes #650
• Fix project operator removing scoped blank nodes, solid/query-ldflex#64

v1.12.0 - 2020-04-03

Changed

• Update to JSON-LD parser to 1.1

Added

• Handle JSON-LD via JSON extension types as well
• Pass HTTP headers to HTML script parsers
• Allow HTML scripts to be targeted by id via fragments
• Throw error on application/json without valid JSON-LD link header

Fixed

• Fix HTML parser not forwarding errors within listeners

v1.11.1 - 2020-04-02

Added

• Add Runner.collectActors method

Fixed

• Fix initialBindings not applied on full query

Changed

• Output number of HTTP requests in stats serializer
• Remove unneeded action argument in Setup.instantiateComponent

v1.11.0 - 2020-03-30

Added

• Add more efficient optional actor
• Expose SPARQL service description in HTTP tool, Closes #219

Changed

• Optimize default graph handling in QPF sources
• Scope blank nodes to each federated source

Fixed

• Fix application aborting on some optional queries
• Use xsd:dateTime format (ISO 8601) for NOW
• Fix several issues with SPARQL endpoints:
  • Fix crash when slicing construct queries
  • Fix only SELECT queries being handleable by SPARQL endpoints
  • Fix SPARQL endpoint detection failing when SD uses blank node subjects #619
  • Fix SPARQL endpoint detection failing for relative IRIs #619
  • Consider sources ending with '/sparql' as SPARQL endpoints, Closes #535

v1.10.0 - 2019-12-12

Added

• Add actors for joining more than two streams pair-wise
• Allow quad-pattern-level contexts to be defined

Changed

• Manually load sources for federated metadata
• Clarify dereferencing error message.

Fixed

• Fix unresolved promise rejections in SPARQL tree serialization
• Swap node-web-streams package with web-streams-node
• Remove unused jsonld dependency

v1.9.4 - 2019-10-21

Changed

• Bump to Components.js 3.3.0

v1.9.3 - 2019-10-16

Added

• Add lenient mode for HTTP and parsing errors
• Store currently processing operation in context
• Log requesting URLs in node-fetch HTTP actor

Changed

• Tweak priorities of default RDF media types
• Make hypermedia link following lazy
• Allow headers and method to be passed to RDF dereferencers
• Update to Sparqlalgebra.js 2.x.x

Fixed

• Add missing stream-to-string to actor-rdf-parse-jsonld

v1.9.2 - 2019-09-27

Added

• Plug HTTP bus into JSON-LD parser, Closes #478
• Detect JSON-LD contexts in link headers, Closes #486
• Throw error when multiple JSON-LD link headers to contexts were detected

Changed

• Emit missing metadata as null
• Remove deprecated util.inspect calls, Closes #488

Fixed

• Fix QPF sources never going beyond page 2
• Reimplement MINUS operator to handle variables according to spec
• Ensure unique blank nodes across different bindings
• Fix sequence path operator actor missing a join mediator
• Filter quads from quad patterns with shared variables
• Fix SPARQL BASE not propagating through operators
• Fix actor-init-sparql bin being broken, Closes #493
• Fix various small Components.js naming issues
• Update sparqlee to 1.2.0 to fix precision errors in basic arithmetics

v1.9.1 - 2019-07-30

Fixed

• Fix file source not emitting metadata, resulting in slow queries
• Fix crash due to missing version bump in rdf-terms

v1.9.0 - 2019-07-29

Added

• Add HTML script tag parser
• Plug RDFa parse into HTML parsing bus, Closes #138
• Add RDF RDFa parser for XML documents
• Add HTTP proxy actor, Closes #465
• Allow actors to be registered before other actors in buses
• Add abstract GraphQL-LD-based hypermedia extractor
• Add query-based metadata extractors, Closes #115
• Add BusIndexed that indexes actors by type

Fixed

• Fix broken docker images, Closes #469
• Fix hashing breaking with different data factories, #438
• Fix GraphQL-LD query results not singularizing
• Fix RDF dereferencing not taking into account relative response URLs

Changed

• Add RDFJS source actor to default SPARQL config
• Bump sparqlee to 1.1.0, Closes #477
• Allow auto-sources to be defined as strings instead of objects
• Add hypermedia-based SPARQL resolver using service descriptions
• Log identified source
• Improve error message on RDF dereference HTTP errors
• Allow combine union mediator to have an empty bus
• Pass headers to RDF parse actors
• Append '/' in accept header when shortened, Closes #471
• Deprecate triple predicate metadata identifier
• Add config option to allow metadata to also be emitted as data
• Add fallback metadata actor that sees everything as both meta and data
• Make primary topic actor fallback to 'all data is also metadata'
• Remove heuristical source identification
• Emit dataset id in Hydra controls extractor
• Rewrite quad pattern hypermedia resolver to be less restrictive
• Add hypermedia resolver for unknown sources
• Refactor QPF-specific hypermedia resolver, #432
• Remove lodash.map from mediators
• Improve performance of applyInitialBindings
• Index the query operation bus
• Remove unneeded string trimming in SPARQL serialization

v1.8.0 - 2019-06-13

Changed

• Update sparqlee to 1.0.0
• Bump to new GraphQL-LD version

v1.7.4 - 2019-05-29

Fixed

• Fix RDF source identifier joining content types incorrectly
• Allow JSON-LD context to appear out-of-order

v1.7.3 - 2019-05-09

Fixed

• Fix bnodes in CONSTRUCT not working, Closes #449
• Fix CONSTRUCT on empty WHERE clause producing no 1 result, Closes #448

v1.7.2 - 2019-05-03

Fixed

• Remove console.log in HTTP parse RDF dereference actor

v1.7.1 - 2019-05-03

Fixed

• Restrict accept headers to 128 characters in browsers to fix CORS issues
• Fix native HTTP actor not forwarding request errors, Closes #381

Changed

• Rescale default media type priorities
• Remove unneeded spaces and zeros in accept headers
• Add CORS headers for SPARQL interface

v1.7.0 - 2019-04-11

Added

• Allow expressions in EXISTS
• Add support for SPARQL aggragates
• Allow baseIRI to be defined for queries

Fixed

• Fix local file querying not working

v1.6.6 - 2019-04-03

Changed

• Remove jsonld.js in favor of streaming JSON-LD processing, Closes #384
• Use real-world endpoint examples in query help message
• Return error code when config compilation failed, Closes #394
• Make SPARQL endpoint identifier use GET instead of HEAD
• Lower default tree serialization priority

Fixed

• Fix bug in orderby due to multiple transforms on same stream, Closes #434
• Fix URL hashes not being stripped when doing HTTP requests, Closes #277
• Fix invalid default REDUCED cache size, Closes #376
• Fix HEAD requests on HTTP service not working, Closes #417
• Fix patterns with conflicting variables being created in federation

v1.6.5 - 2019-03-15

Added

• Emit bound pattern metadata in BGP subcontexts

Fixed

• Fix invalid expression actor test functions

v1.6.4 - 2019-03-05

Fixed

• Fix HTTP service imports

v1.6.3 - 2019-02-27

Changed

• Fix performance issue due to many Error instantiations
• Disable empty sources check in federated actor

v1.6.2 - 2019-02-26

Added

• Add option to invalidate cache before each query exec

Changed

• Improve help message of HTTP tool

v1.6.1 - 2019-02-25

Fixed

• Fix http tool not working with absolute config paths

v1.6.0 - 2019-02-22

Added

• Expose invalidateHttpCache method on SPARQL init
• Support source identification for single sources
• Pass parent metadata to BGP and pattern actions

Changed

• Update sparqlee to version 0.1.0, improves FILTER expressivity
• Use LRU cache in file resolver
• Abstract SPARQL HTTP service

Fixed

• Fix config compilation issues in file init actor
• Fix parser/serializer priority values not being applied
• Fix not all media types being determined in RDF dereference, Closes #318
• Fix invalid unscoped JSON-LD objects

v1.5.4 - 2019-01-31

Fixed

• Fix newEngineDynamic failing to produce results after multiple calls
• Fix http tool not working with absolute config paths

Changed

• Update jsonld to 1.5.0

v1.5.3 - 2019-01-24

Fixed

• Fix GraphQL-LD queries not producing results anymore
• Fix schema and skos prefix

v1.5.2 - 2019-01-17

Fixed

• Fix extend operator still depending on old sparqlee version

v1.5.1 - 2019-01-17

Fixed

• Fix tslib dependency issue in sparqlee, causing a crash at startup
• Fix JSON-LD serializer producing invalid JSON
• Check HTTP stream type when identifying source
• Remove unneeded filter peer dependency from orderby actor
• Remove unneeded @types/bluebird dependency, Closes #360

v1.5.0 - 2019-01-02

Added

• Add Filter operator (based on Sparqlee)
• Add Extend operator (based on Sparqlee)
• Update LeftJoin NestedLoop actor to use Sparqlee
• Add BGP join optimizer, Closes #90
• Add query operation optimize bus

v1.4.6 - 2018-12-11

Fixed

• Fix missing dependencies between new hypermedia actors

v1.4.5 - 2018-12-11

Fixed

• Fix blank nodes in data being seen as variables
• Fix queries with a variable graph only matching the default graph
• Fix missing base IRI in JSON-LD parser, Closes #342

Changed

• Split resolve quad pattern actor into two separate modules

v1.4.4 - 2018-11-13

Fixed

• Update to Components.js 3.2.0
• Upgrade asynciterator to v2.0.1

v1.4.3 - 2018-11-09

Changed

• Update to generic RDFJS typings
• Update N3 to 1.0.0, Closes #314

v1.4.2 - 2018-11-05

Fixed

• Pass baseIRI to RDF parsers, Closes #311

v1.4.1 - 2018-10-04

Fixed

• Make number mediator pick the first when all are invalid

v1.4.0 - 2018-10-03

Added

• Add MINUS operator actor
• Add preconfigured prefixes, Closes #80
• Also check extensions when no valid media type is present, Closes #286
• Make JSON-LD parser accept application/json
• Update sparqljson-parse to 1.4.0 with fix for Virtuoso, Closes #278

Fixed

• Fix federated actor stringifying sources, Closes #265
• Fix federation actor occasionally failing to produce results
• Fix crash when passing raw context with GraphQL, Closes #293
• Fallback to file-based sources when no content type is present, #286
• Fix parsing errors with datetime on resources without Memento support

1.3.0 - 2018-09-10

Added

• Add RDF/XML parser
• Enable serialization in actor-init-sparql-rdfjs

1.2.2 - 2018-09-05

Fixes

• Bump sparqljson-to-tree to 1.2.2

1.2.1 - 2018-09-05

Fixes

• Bump dependencies with invalid tslib dependency
• Add missing actor-query-operation-service dependencies
• Update peer dependency version

1.2.0 - 2018-09-05

Added

• Path expressions
• GraphQL-LD queries
• Memento support with the -d command line option
• SERVICE clause
• Dockerfile
• Browser script uploads to rdfjs/comunica-browser
• RDFJS sources
• Make browser version of the SPARQL init actor consistent with Node
• Allow initial bindings to be passed to query context
• Dereferenceable config instances
• Modularization of configs into config sets
• Action contexts
• Allow actions to be passively observed
• Optimize single SPARQL endpoint sources
• Set user agent for outgoing HTTP requests
• Add logger
• Add federated actor to default HDT init config

Fixes

• Allow pipeline mediator to have no actors in its bus
• Accept falsy terms in quad pattern RDF resolvers
• Fix falsy term crash when querying QPF interfaces
• Update asynciterator and follow-redirects to fix termination problems, Closes #144
• Make empty BGPs return a single empty bindings

1.1.2 - 2018-06-26

Added

• Created actor to read local files

Fixes

• Fix no client-side filtering of non-matching quads, Closes #152
• Update HDT to 2.x.x

1.1.1 - 2018-06-01

Fixes

• Fix config files missing from some init actor after publication

1.1.0 - 2018-06-01

Added

• Add --version flag to command, Closes #118
• Add Pipeline Combine mediator
• Add context preprocessing bus
• Add File RDF Source Identifier
• Add Hypermedia RDF Source identifier
• Add SPARQL RDF Source Identifier
• Add RDF Source Identifier actor as context preprocessor
• Add separate SPARQL init file actor
• Add separate SPARQL init HDT actor

Fixes

• Fix HTTP service not properly recognizing SPARQL POST content types
• Make spec-incompliant expression evaluator handle lang() more loosely
• Fix ActorHttpNative compilation errors on some platforms
• Don't emit null languages in basic expression evaluator
• Catch engine init errors in HTTP service
• Fix incorrect merge of FROM and FROM NAMED graphs
• Loosen handling of undefineds in expression evaluator
• Convert blank nodes when resolving SPARQL patterns, Closes #104
• Make N3 parser and serializer forward error events, #103
• Delegate stream errors from serializers to root, #103
• Make primary topic actor order-independent and take into account void:subset
• Fix paged RDF dereferencer not forwarding triples flag
• Bump Components.js to 2.2.0 to fix config compilation issues on Windows
• Prevent end event from firing before metadata

1.0.4 - 2018-03-22

Fixes

• Fix static newQuery function failing

1.0.3 - 2018-03-22

Fixes

• Catch failure to resolve federated metadata, #117

1.0.2 - 2018-03-22

Fixes

• Fix file quad resolver actor not included as dependency, Closes #120

1.0.1 - 2018-03-21

Fixes

• Add file quad pattern resolver to default config

[1.0.0] - 2018-03-19

• Initial release