=head1 NAME 

Data::FastPack - FastPack Record format, parsing and serialising module

=head1 DESCRIPTION 

Implements an incremental parser to parse an incoming buffer into messages.
Provides basic encoding and decoding functions.

=head1 FASTPACK FORMAT SUMMARY

FastPack format is a binary format for storing records of opaque data related
to a time index into a stream, padded to a multiple of 89 bytes. Within each
stream of data, an ID refers to a channel of data. All multi byte fields are in
little endian byte order processing unless otherwise indicated by the
description/metadata. The fields
of a message are:

  time(double float)  # 8 byte aligned
  id(32)              # uint32
  len(32)	            # uint32
  payload		bytes     # 8 byte aligned
  padding (as required)
  

C<time> is the absolute time of the sample or the difference in time from the
current message to the previous message in the same stream. The exact meaning
of the time is as per the definition messages. It is a double float to allow web
browsers to utilise high resolution time, as they do not support 64bit integers.


C<id> is is the channel id within the file/stream. It relates to a definition
file. 0 indicates a meta data point which is JSON or other structured data,
which alters the processing of the file. 

C<len> is the length of the payload. If the length is larger than 2^32 then it
must be fragmented at the 'application level'.


C<payload> is the data.

C<padding> Every record is padded to an 8 byte bounadary, with nulls, if nessicary



=head1 CONCEPTS 

The message format is primarily intended to store a sequence of time indexed
values which are to be parameterized to another channel.  For example indexing
a sensor to position where both the sensor and the position are sampled
separately, but can be stored with the same time base.

There is only one reserved channel id, 0 , which is a meta data channel. This
channel is JSON or message pack, and provides the meta/header data to control
the stream from that point forward.

The meta data semantics are application dependant, giving great flexibility, in
time base, channel relationships etc

=over 

=item * Efficient use of memory access for ARM cpus

Multi byte data types are stored in little endian order, unless otherwise
specified in the meta data. Payloads are also on a 8 byte boundary allows
direct access for double precision float


=item * Messages stored in one or more files

Multiple streams of message can be stored in a single file if they share a time
base. For streams that have differing time bases, they are stored in a separate
files. This give good compression ability

=item * External defintition file(s) for message types if required.

The definitions of a file can be pointed to externally, or can be stored
internally in a meta data message

=item * Highly compresssable and suitable for self contained web applications

The message time, id, length fields will be mostly unchanging when multiple
message source of same time base are recorded together. The 24 byte header will
basically be reduced to 1 or 2 bytes after compression for most messages.

=back



=head2 Timimg

Timing data is a double float field and can represent many different timing
scenarios.
	
=over

=item * Direct time (seconds)

The simplest case is the storing seconds as floating point values in the field.
Whether the value is a difference to the previous message or an absolute value
is based on definition messages for the file.

=item * Multiple of a time base

Similar to direct time above, however the value is multiplied by an external
time base factor to generate the actual time.

=item * Argument/index into a timing function.

The value is used as an index into a timing function stored in external
JavaScript, which when called calculates a time. ie for processing video with
fixed non integer frame rates

=back

For a system reporting only a single message, the time field will constantly be
updated for each message. However with multiple channels (eg, gamma, caliper,
lsd, ssd), only the first message from the group will have a non zero time when
using difference mode.  Most messages in a system will have a 0 time because of
this. 

=head2 Padding

Padding to an 8 byte boundary is implicit to every message.  Arbitrary bytes
can be appended to ensure the alignment.


=head2 Payload Length

A 32 bit field indicating the length of the data in the payload. It is stored
just before the payload to allow more efficient decoding in scripting languages

=head2 Payload

The payload of the message. It is 8 byte aligned for better memory access (ie a
double can extracted directly out of the payload field)


=head1 META AND STRUCTURED DATA

All message ids of 0 are designated "Meta Data". This means the
payload is encoded either in a JSON array or object or as a MessagePack
structure


This gives the fast encoding/decoding of simple time series values and the
ability to have arbitrarily complex data when required

Decoding of meta data automatically picks MesagePack or JSON as required,
as long as the encoded values are are map or array types.

It is recommended that general structured data in other message ids also either
JSON or MessagePack also.

=cut

