The Javascript engine can be used in a modern browser or a server engine like
Node.js.
The Javascript clone engine conforms to the m-ldspecification. Its support for transaction patterns
is detailed below. Its API concurrency model
is based on serialised consistent states.
The live code app below creates a domain of shared information, and then follows updates to it. In its user interface, it just presents the domain name. (You can see and edit the code by clicking the button, left-middle.)
The new domain's information is stored in memory here (and only here). The next live code app, below, allows us to make another clone of the domain. These two code blocks are sandboxed – they do not have access to each other's state via this browser.
💡 You can confirm this by trying it in another window (use the button, top right), another browser, or another device... anywhere in the world!
The domain is using a public Gateway (gw.m-ld.org) to connect clones together. This means the m-ld playground can also see it.
💡 m-ld domain names look like IETF internet domains, and have the same rules. The internet doesn't know how to look them up yet though, so you can't just paste one into a browser.
There are more live code examples for you to try in the How-To section below.
For a scalable global managed service, use AblyRemotes.
If you have a live web server (not just CDN or serverless), you can use
IoRemotes.
💡 If your architecture includes some other publish/subscribe service like AMQP or Apache Kafka, or you would like to use a fully peer-to-peer protocol, please contact us to discuss your use-case. Remotes can even utilise multiple transport protocols, for example WebRTC with a suitable signalling service.
The clone function returns control as soon as it is safe to start making data transactions against the domain. If this clone has been re-started from persisted state, it may still be receiving updates from the domain. This can cause a UI to start showing these updates. If instead, you want to wait until the clone has the most recent data, you can add:
MQTT is a machine-to-machine (M2M)/"Internet of Things"
connectivity protocol. It is convenient to use it for local development or if
the deployment environment has an MQTT broker available. See below for specific
broker requirements.
The MqttRemotes class and its companion configuration class MeldMqttConfig
can be imported or required from '@m-ld/m-ld/ext/mqtt'. You must also
install the async-mqtt package as a peer of @m-ld/m-ld.
A bundle is also available at http://js.m-ld.org/ext/mqtt.mjs.
The configuration interface adds an mqtt key to the base
MeldConfig. The content of this key is a client
options object for MQTT.js. It must
not include the will and clientId options, as these are set internally. It
must include a hostnameor a host and port, e.g.
MQTT remotes supports websockets for use in a browser environment. To configure,
add protocol: 'ws' (or 'wss') to the mqtt configuration value. (Note that
All the MQTT configuration goes through the mqtt key, even if it's actually
using websockets for transport.) This requires the MQTT broker to support
websocket connections, for example see the
Aedes documentation.
Ably provides infrastructure and APIs to power realtime
experiences at scale. It is a managed service, and includes pay-as-you-go
developer pricing. It is also convenient to use
for global deployments without the need to self-manage a broker.
The AblyRemotes class and its companion configuration class MeldAblyConfig
can be imported or required from '@m-ld/m-ld/ext/ably'. You must also
install the ably package as a peer of @m-ld/m-ld.
A bundle is also available at http://js.m-ld.org/ext/ably.mjs.
The configuration interface adds an ably key to the base
MeldConfig. The content of this key is an Ably
client options
object. It
must not include the echoMessages and clientId options, as these are set
internally.
If using token
authentication,
ensure that the clientId the token is generated for corresponds to the @id
given in the MeldConfig.
Socket.IO enables real-time, bidirectional and event-based
communication. It works on every platform, browser or device, focusing equally
on reliability and speed. It is convenient to use when the app architecture has
a live web server or app server, using HTTP.
The IoRemotes class and its companion configuration class MeldIoConfig
can be imported or required from '@m-ld/m-ld/ext/socket.io'. You must also
install the socket.io-client package as a peer of @m-ld/m-ld.
A bundle is also available at http://js.m-ld.org/ext/socket.io.mjs.
The configuration interface adds an io key to the base
MeldConfig. The value is an optional object
having:
uri: The server URL (defaults to the browser URL with no path)
opts: A
Socket.io factory options
object, which can be used to customise the server connection
When using Socket.io, the server must correctly route m-ld protocol
operations to their intended recipients. The Javascript engine package bundles a
class for Node.js servers, IoRemotesService, which can be imported
from '@m-ld/m-ld/ext/socket.io/server'.
To use, initialise the
Socket.io server as normal, and then construct an IoRemotesService, passing
the namespace you want to make available to m-ld. To use the global
namespace, pass the sockets member of the Server class. For example:
const socket = require('socket.io');
const httpServer = require('http').createServer();
// Start the Socket.io server, and attach the m-ld message-passing serviceconst io = new socket.Server(httpServer);
new IoRemotesService(io.sockets);
Group or Subject (the shorthand way to insert data)
Update (the longhand way to insert or delete data)
💡 If you have a requirement for an unsupported pattern, please
contact us to discuss your use-case. You can
browse the full json-rql syntax at
json-rql.org.
Subjects in the Javascript engine are accepted and presented as plain Javascript objects whose content is JSON-LD (see the m-ldSpecification). Utilities are provided to help the app use and produce valid subjects.
includeValues includes one or more given value in a subject property.
includesValue determines whether the given subject property contains the given value.
propertyValue gets a property value from the given subject, casted as a strict type.
Clone updates obtained from a read handler specify the exact Subject property values that have been deleted or inserted during the update. Since apps often maintain subjects in memory, for example in a user interface, utilities are provided to help update these in-memory subjects based on updates:
updateSubject applies an update to the given subject in-place.
SubjectUpdater applies an update to more than one subject.
asSubjectUpdates provides an alternate view of the update deletes and inserts, by Subject, for custom processing.
A m-ld clone contains realtime domain data
in principle. This means that any clone
operation may be occurring concurrently
with operations on other clones, and these operations combine to realise the
final convergent state of every clone.
The Javascript clone engine API supports bounded procedures on immutable state,
for situations where a query or other data operation may want to operate on a
state that is unaffected by concurrent operations. In general this means that in
order to guarantee data consistency, it is not necessary for the app to use the
clone's local clock ticks (which nevertheless appear in places in the API for
consistency with other engines).
An immutable state can be obtained using the
read and
write methods. The state is passed to a
procedure which operates on it. In both cases, the state remains immutable until
the procedure's returned Promise resolves or rejects.
In the case of write, the state can be transitioned to a new state from within
the procedure using its own write method,
which returns a new immutable state.
In the case of read, changes to the state following the procedure can be
listened to using the second parameter, a handler for new states. As well as
each update in turn, this handler also receives an immutable state following the
given update.
await clone.read(async (state: MeldReadState) => {
// State in a procedure is locked until sync complete or returned promise resolveslet currentData = await state.read(someQuery);
populateTheUi(currentData);
}, async (update: MeldUpdate, state: MeldReadState) => {
// The handler happens for every update after the proc// State in a handler is locked until sync complete or returned promise resolves
updateTheUi(update); // See §Handling Updates, below
});
ui.on('action', async () => {
clone.write(async (state: MeldState) => {
let currentData = await state.read(something);
let externalStuff = await doExternals(currentData);
let todo = decideWhatToDo(externalStuff);
// Writing to the current state creates a new live state
state = await state.write(todo);
await checkStuff(state);
});
});
These examples show simple patterns for getting started with an app's code structure. In principle, m-ld acts as a "model", replacing (or being proxied by) the local in-memory data model. Because m-ld information is fundamentally live – it can change due to a remote edit as well as a local one – it's valuable for the local app code to react to changes that it may not have initiated itself.
One of the most common questions asked about live information models is, what happens if there is a "conflict"? Here, we handle one particular kind of conflict using declarative constraints.
By default, information in m-ld is an unordered graph (just like in a relational database). This example shows how ordered lists can be embedded in the graph.
Text embedded in a structured graph of information might need to be editable by multiple users at the same time, like an online document. This example shows the use of an embedded data type and a supporting HTML control, to enable multi-player text editing.
The information in m-ld is stored using the W3C standard data representation RDF (Resource Description Framework). For RDF-native apps, the Javascript engine API supports direct access to the RDF graph.
To mitigate integrity, confidentiality and availability threats to m-ld domain data, we recommend the following baseline security controls for your app.
Apply authorisation controls to remote connections, restricting access to authorised users of the app. This typically requires users to be authenticated in the app prior to connecting the m-ld clone. Authentication credentials can be passed to the remotes implementation via the configuration object.
Apply operating-system access control to the storage location used for persistence, restricting access to authorised users of the app.
Encrypt remote connection transports, for example by applying transport layer security (TLS) to server connections, if applicable.
A more general discussion of security considerations in m-ld can be found on the website.
🧪 This library additionally includes an experimental extension for controlling access based on an Access Control List (ACL) in the domain data. Please see our Security Project for more details of our ongoing security research, and contact us to discuss your security requirements!
m-ld supports extensions to its core engine. You can choose which extensions to use in an app; some are bundled in the engine package, others you can write yourself.
Some extensions must be pre-selected by the app in order to connect a new clone to a domain of information. Other extensions can be declared in the data and loaded dynamically by the engine at runtime. This allows authors to add features to the shared information, without the need for a central authority over features. This is decentralised extensibility, similar to how a web page can declare scripts that extend its features in the browser. See our introductory short paper for more about this vision.
The Javascript engine package bundles the following extensions – follow the links to learn more:
Remotes are pre-selected in the call to clone, as described above.
Lists have a default implementation, which is replaceable.
Shapes are used to enforce a 'schema' or 'object model' on the domain's data.
Text subject properties can be made collaboratively-editable.
Transport Security allows an app to encrypt and apply digital signatures to m-ld protocol network traffic.
Statutes allow an app to require that certain changes, such as changes to access controls, are agreed before they are shared in the domain.
The extension's code module must be available to a global CommonJS-style require method in all clones using the Javascript engine. For bundled extensions:
In Node.js, the module is packaged in @m-ld/m-ld; no additional configuration is required.
In the browser, require is typically provided by the bundler. Since the module will be loaded dynamically, the bundler may need to be configured to guarantee the module is bundled, since it may not be referenced statically by any code.
💡 While it's possible to change extensions at runtime (by changing their declarations in the data), this may require coordination between clones, to prevent outdated clones from acting incorrectly in ways that could cause data corruption or compromise security. Consult the extension's documentation for safe operation.
Extension code is executed as required by the core engine or by another extension. Besides remotes, there are currently four types of custom extension called by the core engine, defined in the MeldExtensions API interface. To write an extension to the core, you must implement one or more of these types.
💡 Please do contact us if you would like to understand more about extensions.
Configuration of the clone data constraint. The supported constraints are:
single-valued: the given property should have only one value. The
property can be given in unexpanded form, as it appears in JSON subjects
when using the API, or as its full IRI reference.
Initial definition of a m-ld app. Extensions provided will be used for
bootstrapping, prior to the clone joining the domain. After that, different
extensions may come into effect if so declared in the data.
A function type to find the correct Datatype for an identifier and
optionally a property in the domain.
If property is provided, datatype is the datatype of a literal at the
given property position in a Subject. Otherwise, it is the identity of the
datatype itself (which may be the same).
A subscription to a state machine. Can be unsubscribed to stop receiving
updates. The subscription itself can also be async-iterated. Finally, the
subscription may have a resolved value that can be awaited.
When used as an async iterable, it's important to begin iteration
synchronously in order not to miss any updates. It is safe to await the
subscription resolved value, if applicable – but it's rare to need both the
resolved value and iteration.
When used as a promise, calling unsubscribe before the promise is settled may
cause it to reject with EmptyError.
A function type specifying a 'procedure' during which a clone state is
available as immutable. Strictly, the immutable state is guaranteed to remain
'live' until the procedure's return Promise resolves or rejects.
An update form that mirrors the structure of a GraphUpdate, having
optional keys
Type parameters
T
UpdateProc
UpdateProc<U, T>:(update: U, state: MeldReadState) => PromiseLike<T> | T
A function type specifying a 'procedure' during which a clone state is
available as immutable following an update. Strictly, the immutable state is
guaranteed to remain 'live' until the procedure's return Promise resolves or
rejects.
An operator-based constraint of the form { <operator> : [<expression>...] }. The key is the operator, and the value is the array of arguments. If the
operator is unary, the expression need not be wrapped in an array.
A JSON-LD context for some JSON content such as a Subject. m-ld
does not require the use of a context, as plain JSON data will be stored
in the context of the domain. However in advanced usage, such as for
integration with existing systems, it may be useful to provide other context
for shared data.
A reference to a Subject. Used to disambiguate an IRI from a plain string.
Unless a custom Context is used for the clone, all references
will use this format.
This type is also used to distinguish identified subjects (with an @id
field) from anonymous ones (without an @id field).
'Properties' of a Subject, including from List and Slot.
Strictly, these are possible paths to a SubjectPropertyObject
aggregated by the Subject. An @list contains numeric indexes (which may be
numeric strings or variables). The second optional index is used for multiple
items being inserted at the first index, using an array.
The allowable types for a Subject property value, named awkwardly to avoid
overloading Object. Represents the "object" of a property, in the sense of
the object of discourse.
An operation against the TSeq data types comprises a set of runs.
experimental
TSeqRevert
TSeqRevert:(TSeqCharTick | undefined)[][]
A revert of a TSeq operation encodes the prior state of each char-tick in an
operation. It has the same length as its corresponding operation, where each
index in the outer array matches an operation run. An undefined entry
indicates that no change happened for that char-tick.
experimental
TSeqRun
TSeqRun:[TSeqName[], TSeqCharTick[]]
A 'run' is a sequence of affected characters (content) at an anchor position
in the tree identified by a path, which is an array of names.
Create or initialise a local clone, depending on whether the given backend
already contains m-ld data. This function returns as soon as it is safe to
begin transactions against the clone; this may be before the clone has
received all updates from the domain. You can wait until the clone is
up-to-date using the MeldClone.status property.
isPropertyObject(property: string, object: unknown): object is SubjectPropertyObject
Determines whether the given property object from a well-formed Subject is a
graph edge; i.e. not a @context or the Subject @id.
Parameters
property: string
the Subject property in question
object: unknown
the object (value) of the property
Returns object is SubjectPropertyObject
isRead
isRead(p: Pattern): p is Read
Determines if the given pattern will read data from the domain.
Parameters
p: Pattern
Returns p is Read
isWrite
isWrite(p: Pattern): p is Write
Determines if the given pattern can probably be interpreted as a logical
write of data to the domain.
This function is not exhaustive, and a pattern identified as a write can
still turn out to be illogical, for example if it contains an @insert with
embedded variables and no @where clause to bind them.
Returns true if the logical write is a trivial no-op, such as {},
{ "@insert": {} } or { "@graph": [] }.
array<T>(value?: T | T[] | null): NonNullable<T>[]
Utility to normalise a property value according to m-lddata semantics, from a missing
value (null or undefined), a single value, or an array of values, to an
array of values (empty for missing values). This can simplify processing of
property values in common cases.
In many cases it is preferable to apply inserted and deleted properties to
app data views on a subject-by-subject basis. This property views the above
as:
noMerge<T>(type: AtomType<T>, ...values: ValueConstructed<T>[]): never
An atom value merge strategy that refused to merge and throws. This should be
used in situations where a exception is suitable for the application logic.
Note that in many situations it may be better to declare the property as an
Array or Set, and to present the conflict to the user for resolution.
A deterministic refinement of the greater-than operator used for SPARQL
ordering. Assumes no unbound values, blank nodes or simple literals (every
literal is typed).
updateSubject<T>(subject: T, update: SubjectUpdateLike, ignoreUnsupported?: boolean): T
Applies an update to the given subject in-place. This method will correctly
apply the deleted and inserted properties from the update, accounting for
m-lddata semantics.
Referenced Subjects will also be updated if they have been affected by the
given update, deeply. If a reference property has changed to a different
object (whether or not that object is present in the update), it will be
updated to a json-rql Reference (e.g. { '@id': '<iri>' }).
Changes are applied to non-@list properties using only L-value assignment,
so the given Subject can be safely implemented with property setters, such as
using set in a class, or by using defineProperty, or using a Proxy; for
example to trigger side-effect behaviour. Removed properties are set to an
empty array ([]) to signal emptiness, and then deleted.
Changes to @list items are enacted in reverse index order, by calling
splice. If the @list value is a hash, it will have a length property
added. To intercept these calls, re-implement splice on the @list
property value.
Changes to text represented as a shared datatype
supporting the @splice operator will be correctly applied to plain strings,
and can also be applied to any Object having a splice method taking three
parameters, index, deleteCount and content. This allows an app to
efficiently apply character-level changes, e.g. to document editing
components. The proxy object should also implement a toJSON() method which
returns the current string state.
CAUTION: If this function is called independently on subjects which reference
each other via Javascript references, or share referenced subjects, then the
referenced subjects may be updated more than once, with unexpected results.
To avoid this, use a SubjectUpdater to process the whole update.
if false, any unsupported data expressions in the
update will cause a RangeError – useful in development to catch problems
early
Returns T
uuid
uuid(): string
Utility to generate a unique short UUID for use in a MeldConfig; actually
a CUID starting with the character c and containing only lowercase
US-English letters and digits. (Note that this is not an RFC 4122 UUID.)
Configuration of the clone data constraint. The supported constraints are:
single-valued: the given property should have only one value. The property can be given in unexpanded form, as it appears in JSON subjects when using the API, or as its full IRI reference.See Shape