The DOOD library#

The Dylan Object Oriented Database#

The Dylan Object Oriented Database (DOOD) is a simple mechanism for storing arbitrary objects and lazily loading them. During dump time, DOOD traverses a graph of objects and encodes the objects as a sequence of bytes. These bytes are later interpreted by DOOD during load time reconstructing an isomorphic object-graph in which cyclic structures and shared references are preserved. DOOD provides ways to control what slots in objects should be stored, and to decide what objects should be stored by proxies.

DOOD was meant to be a very simple Dylan object store that supported:

  • pay as you go

  • incremental loading,

  • flexible proxies, and

  • non-corrupting commits

DOOD was not meant to provide:

  • multiuser support,

  • full blown transaction support,

  • incremental writes,

  • schema evolution, nor

  • client/server support

In order to add persistence to a program, the easiest thing is to just save and load the data completely:

define method dump-data (data, locator)
  let dood = make(<dood>, locator: locator, direction: #"output",
                  if-exists: #"replace");
  dood-root(dood) := data;
  dood-commit(dood);
  dood-close(dood);
end method;

define method load-data (locator) => (data)
  let dood = make(<dood>, locator: locator, direction: #"input");
  let data = dood-root(dood);
  dood-close(dood);
  data
end method;

This works great for simple applications. More complicated applications potentially require support for data compression, special reinitialization, lazy loading of data, and multiple databases. Data can be compressed by being able to specify that only a subset of an object’s slots are saved (for example, an object may cache information in a slot) or more generally by dumping the object’s information in a completely different format on disk. The former technique is described in the schema section below and the latter technique is described in the proxy section below.

The DOOD module#

<dood> Open Primary Class#
Superclasses:

<object>

Init-Keywords:

  • if-exists – An instance of <object>. Specifies the actions to take during creation of a file stream when the locator: init-keyword is specified. If if-exists: init-keyword specifies #"replace" then the database is considered empty.

  • backups? – An instance of <boolean>. Specifies whether a new file is used during a commit. backups? is always false when the stream: init-keyword is specified.

  • batch-mode? – An instance of <boolean>.

  • default-segment – An instance of <dood-segment>.

  • locator – An instance of <object>. Should be a object valid for opening a stream with a locator: init-keyword.

  • name – An instance of <object>. Usually a string or symbol but user-controlled.

  • read-only? – An instance of <boolean>.

  • segments – An instance of <simple-object-vector>.

  • stream – An instance of false-or(<stream>).

  • version – An instance of <integer>. Specifies a version that can be used for version control. Upon opening of an existing database, the specified version is compared against the stored version, and if they are different a <dood-user-version-warning> condition is signaled.

  • world – An instance of <dood-world>.

This class can be user subclassed and used as the basis for specialized loading and dumping behavior.

<dood-opening-warning> Class#
Superclasses:

<dood-warning>

Init-Keywords:

  • dood (required) – An instance of <dood>.

Superclass of all dood warnings.

<dood-corruption-warning> Class#
Superclasses:

<dood-opening-warning>

Signaled if DOOD data is found to be corrupted.

<dood-version-warning> Class#
Superclasses:

<dood-opening-warning>

Signaled if the current DOOD version is different from the saved DOOD version.

<dood-user-version-warning> Class#
Superclasses:

<dood-version-warning>

Signaled if the specified user version is different from the saved user version.

dood-name(<dood>) Method#

Returns the name of the specified dood.

dood-root Generic function#
Signature:

dood-root (object) => (value)

Parameters:

  • object – An instance of <dood>

Values:

  • value – An instance of <object>.

Returns the one distinguished root of the specified dood. It is defaulted to be #f when a new dood is created.

dood-root-setter Generic function#
Signature:

dood-root-setter (value object) => (value)

Parameters:

  • value – An instance of <object>.

  • object – An instance of {<dood> in dood}.

Values:

  • value – An instance of <object>.

Sets the one distinguished root of the specified dood.

dood-commit Generic function#
Signature:

dood-commit (dood #key flush? dump? break? parents? clear? stats? size) => ()

Parameters:

  • dood – An instance of <dood>.

  • flush? (#key) – An instance of <object>.

  • dump? (#key) – An instance of <object>.

  • break? (#key) – An instance of <object>.

  • parents? (#key) – An instance of <object>.

  • clear? (#key) – An instance of <object>.

  • stats? (#key) – An instance of <object>.

  • size (#key) – An instance of <object>.

Saves the data reachable from dood-root. When the backups?: init-keyword is specified to be true, the data is first written to a new file. The new file is named the same as the specified locator but with its suffix changed to “new”. Upon success, the original data file is replaced with the new file. Upon failure the new file is just removed and the original data file is untouched.

dood-size Generic function#
Signature:

dood-size (dood) => (res)

Parameters:

  • dood – An instance of <dood>.

Values:

  • res – An instance of <integer>.

Returns the size in bytes of the data file.

dood-close Generic function#
Signature:

dood-close (dood #rest all-keys #key abort?) => ()

Parameters:

  • dood – An instance of <dood>.

  • all-keys (#rest) – An instance of <object>.

  • abort? (#key) – An instance of <object>.

Closes the specified database and underlying file stream if created with locator: init-keyword. If the stream: init-keyword was specified then it is the user’s responsibility to close this stream. Abort is passed through to close an underlying file stream.

Schemas#

Schemas are declarative descriptions of how objects are persistently dumped and loaded. In the current version of DOOD, schemas are not themselves persistently stored. User versions can be used to manually ensure compatible schemas.

dood-class-definer Macro#

The dood-class-definer macro defines a dylan class with extra slot adjectives specifying the dumping and loading behavior of the corresponding slot. The default DOOD treatment of a slot, called deep, is that its contents is recursively dumped and eagerly loaded. There are three dood slot adjectives that modify this behavior: lazy, disk, and weak. A lazy slot’s contents is recursively dumped and lazily loaded, that is, loaded from disk upon first access. A disk slot’s contents is recursively dumped and is always loaded from disk when and only when explicitly accessed and is never written back to the slot. A weak slot’s contents is never dumped and a user can specify a reinit-expression to be used instead during loading. A reinit-expression must be specified even if an init-expression is the same, otherwise reinitialization will not occur and the slot will be unbound. In the current version of DOOD, the reinit-expression must appear as the first slot keyword parameter if at all. Accessing lazy slot values in a closed database will signal a dood-proxy-error (see below).

Example:

define dood-class <computation> (<object>)
  lazy slot computation-source-location :: false-or(<source-location>) = #f,
    init-keyword: source-location:;
  slot computation-previous :: <compution>,
    required-init-keyword: previous:;
  slot computation-next :: <computation>,
    required-init-keyword: previous:;
  weak slot computation-type :: false-or(<type-estimate>) = #f,
    reinit-expression: #f;
end dood-class;

Reading#

Internally DOOD loads objects by instantiation and slot assignment. An object is instantiated via the internal system allocator, which returns an uninitialized instance, and then initialized by applying the setters of an object’s class.

dood-reinitialize Open Generic function#
Signature:

dood-reinitialize (dood object) => ()

Parameters:

  • dood – An instance of <dood>.

  • object – An instance of <object>.

For some objects the simple instantiation and slot assignment approach will not produce a well-formed object. dood-reinitialize gives objects a chance to correct any reconstruction problems. This function is called on an object immediately after the object has been loaded from disk.

Example:

define dood-class <rectangle> (<object>)
  slot rectangle-height :: <integer>,
    required-init-keyword: height:;
  slot rectangle-width :: <integer>,
    required-init-keyword: width:;
  weak rectangle-area :: <integer>;
end dood-class;

define method dood-reinitialize (dood :: <dood>, object :: <rectangle>)
  next-method();
  rectangle-area(object)
    := rectangle-height(object) * rectangle-width(object);
end method;

Tables#

<dood-lazy-symbol-table> Class#
Superclasses:

<dood-lazy-key-table>

Provide a mechanism for indexes. The keys are symbols and are loaded lazily using a binary search. This is known to be an inferior layout strategy and will be replaced by b*-trees in the future.

dood-lazy-forward-iteration-protocol Generic function#
Signature:

dood-lazy-forward-iteration-protocol (table) => (#rest results)

Parameters:

  • table – An instance of <object>.

Values:

  • #rest results – An instance of <object>.

Used for walking only keys presently loaded. The standard forward-iteration-protocol will load all keys and values into memory.

Proxies#

Sometimes users need more control over how objects are dumped to disk. DOOD provide a general mechanism called a proxy, which provides both a disk representation of an object and a reconstruction policy. The basic idea is that during the dumping process each memory object is given a chance to provide a disk object (a proxy) to be used for dumping and then upon loading, a loaded disk object is given a chance to map back to its original memory object. Proxies can be used for mapping objects back to unique runtime objects, for compressing objects, for looking up objects in external databases, etc.

<dood-proxy> Open Class#
Superclasses:

<dood-mapped-and-owned-object>

This is the superclass of all proxy objects. Users must subclass this class in order to define a new kind of proxy.

dood-disk-object Open Generic function#
Signature:

dood-disk-object (dood object) => (disk-object)

Parameters:

  • dood – An instance of <dood>.

  • object – An instance of <object>.

Values:

  • disk-object – An instance of <object>.

Users write methods on this generic when they want an object to have a proxy. It returns a disk-object which is dumped in lieu of the memory-object.

dood-disk-object(<dood>, <object>) Method#
dood-disk-object(<dood>, <dood-mapped-and-owned-object>) Method#
dood-disk-object(<dood>, <generic-function>) Method#
dood-disk-object(<dood>, <function>) Method#
dood-disk-object(<dood>, <class>) Method#
dood-disk-object(<dood>, <integer>) Method#
dood-restore-proxy Open Generic function#
Signature:

dood-restore-proxy (dood proxy) => (memory-object)

Parameters:
Values:

  • memory-object – An instance of <object>.

This function is called immediately after a proxy is reconstructed with instantiation and slot assignment. Its job is to map from a disk-object back to its memory-object.

dood-restore-proxy(<dood>, <dood-program-module-proxy>) Method#
dood-restore-proxy(<dood>, <dood-program-binding-proxy>) Method#
dood-restore-proxy(<dood>, <dood-class-program-binding-proxy>) Method#
<dood-proxy-error> Open Class#
Superclasses:

<error>

Signaled when proxy is restored from closed database.

Proxy examples#

Dump by Reference#

The first example shows how proxies can be used to dump objects by reference back to objects in a user’s program. This is necessary when the data can not be dumped by DOOD (e.g., functions).

define constant $boot-objects = make(<table>);
define class <boot-object> (<object>)
  slot boot-id :: <integer>, required-init-keyword: id:;
  slot boot-function :: <function>, required-init-keyword: function:;
end class;

define method initialize (object :: <boot-object>, #key id, #all-keys)
  next-method();
  $boot-objects[id] := object;
end method;

define class <boot-proxy> (<dood-proxy>)
  slot boot-id :: <integer>, required-init-keyword: id:;
end class;

define method dood-disk-object
    (dood :: <dood>, object :: <boot-object>) => (proxy :: <boot-proxy>)
  make(<boot-proxy>, id: boot-id(object))
end method;

define method dood-restore-proxy
    (dood :: <dood>, proxy :: <boot-proxy>) => (object :: <boot-object>)
  $boot-objects[boot-id(proxy)]
end method;

Compression#

The second example shows how proxies can be used to compress an object’s disk representation.

define class <person> (<object>)
  slot person-gender :: one-of(#"male", #"female"),
    required-init-keyword: gender:;
  slot person-height :: <integer>,
    required-init-keyword: height:;
end class;

define constant <person-code> = <integer>;

define method encode-person (person :: <person>) => (code :: <person-code>)
  if (person-gender(person) == #"male") 0 else 1 end
      + (person-height(person) * 2)
end method;

define method decode-person (code :: <person-code>) => (person :: <person>)
  person-gender(person) := if (even?(code)) #"male" else #"female";
  person-height(person) := truncate/(code, 2);
end method;

define class <person-proxy> (<dood-proxy>)
  slot person-code :: <person-code>, required-init-keyword: code:;
end class;

define method dood-disk-object
    (dood :: <dood>, object :: <person>) => (proxy :: <person-proxy>)
  make(<person-proxy>, code: encode-person(object))
end method;

define method dood-restore-proxy
    (dood :: <dood>, proxy :: <person-proxy>) => (object :: <person>)
  make(<person>, decode-person(person-code(object)))
end method;

Multiple Databases#

The third example demonstrates how to use proxies for interdatabase references. Suppose that each database is registered in a symbol-table of databases and that each object stored in these databases knows both its name and to which database it belongs. Furthermore, suppose that each database has a lazy symbol-table stored as its distinguished root.

define class <dooded-object> (<object>)
  slot object-dood-name,    required-init-keyword: dood-name:;
  slot object-binding-name, required-init-keyword: binding-name:;
  // ...
end class;

define class <dood-cross-binding-proxy> (<dood-proxy>)
  slot proxy-dood-name,    required-init-keyword: dood-name:;
  slot proxy-binding-name, required-init-keyword: binding-name:;
end class;

define method dood-external-object (dood :: <dood>, name :: <symbol>)
  let symbol-table = dood-root(dood);
  element(symbol-table, name, default: #f)
end method;

define constant $doods = make(<table>);

define method lookup-dood (name :: <symbol>) => (dood :: <dood>)
  element($doods, name, default: #f)
    | (element($doods, name)
       := make(<dood>, locator: as(<string>, name), direction: #"input"))
end method;

define method dood-restore-proxy
    (dood :: <dood>, proxy :: <dood-cross-binding-proxy>) => (object)
  let external-dood = lookup-dood(proxy-dood-name(proxy);
  dood-external-object(external-dood, proxy-binding-name(proxy))
end method;

define method dood-disk-object
    (dood :: <dood>, object :: <dooded-object>) => (disk-object)
  if (dood-name(dood) == object-dood-name(object)) // local?
    object
  else
    make(<dood-cross-binding-proxy>,
      dood-name:    dood-name(dood),
      binding-name: object-binding-name(object))
  end if;
end method;