The DMX file format

Other resources:

DMX stands for Data Model eXchange format. It's used in TF2 to store particle systems, those files ending in ".pcf". Here are some informations on the structure of the binary format I was able to get by searching over the Internet and reading the binary files.

Heads-up: This ONLY covers the format of binary files version 2, the format used for TF2's particle files. The format changes in more recent versions (an example is the strings that aren't arrays moving to the dictionary in v4, or the dictionary indices being 4 bytes instead of 2). You can get a list of (some of?) the changes here. The format is also completely different for the ascii version (keyvalues2).

Heads-up: This is ONLY based on my research. This may contain errors.

Overview

Binary files are in little-endian and composed of 4 parts:

The header. It's an string that specifies the encoding ("binary" or "keyvalues2" (text)), encoding version, format (which can be multiple things, like "dmx" or "pcf" (which seems to have no difference in binary 2)) and format version. It's written like this: \n\0. Note that the string finishes with a newline and a null byte.
The string dictionary. This is where (most of) the strings used in the format are stored, to save space. They are referenced by multiple elements/attributes with an index. It starts by 2 bytes containing the number of entries in the dictionary, followed by null-terminated strings.
```
num of entries | string | string | string | ...
      2            N        N        N
```
The "element index". This is where all the elements are stored, one after another. It starts by 4 bytes specifying the number of elements, followed by the following structure: 2 bytes specifying the element's "type" (which is simply an index to an entry in the dictionary, as the types of elements are simply strings), followed by a null-terminated string (not an index!) which is the element's name, then 16 bytes specifying the element's UUID (universally unique identifier, every element possess one.) It seems that the UUID is a version 4, variant 2 one, but I do not know if it has any real use or if UUIDs can be of any version/variant.
```
num of elements | type | name | UUID | type | name | UUID | ...
       4           2      N      16     2      N      16
```
The elements' attributes. The order of the attributes corresponds to the order of the elements, so the first pack of attributes belongs to the first parsed element, etc. Every pack of attributes has the following structure: 4 bytes specifying the number of attributes (for one element), then for each attribute, the name (a 2 bytes index to the dictionary), the type (one byte), then the value (whose length differs depending on the type).
```
num of attributes | name | type | value | name | type | value | ... | num of attributes | name | ...
        4            2      1       N      2      1       N                   2            2
```
All attributes have an array counterpart, whose format is different: instead of the single value, you have: 4 bytes specifying the number of values in the array, then all the values contained in the array. values in the array are always of the same type. The type of the attribute will tell if it's an array or not.
```
... | num of members | value | value | value | ...
            4            N       N       N
```

Here is a hexdump of a very simple DMX file, with the bytes colored to more clearly see what is what.
A hexdump of a PCF file.

This image corresponds to this file (in keyvalues2 format):

<!-- dmx encoding keyvalues2 1 format pcf 1 -->
"DmeElement"
{
	"id" "elementid" "725b1da6-1859-449b-ac0e-c37405bc14ee"
	"name" "string" "Test element"
	"test_float" "float" "2.020004325"
	"test_vec3" "vector3" "1 2.1 3.5"
	"test_bool_on" "bool" "1"
	"test_bool_off" "bool" "0"
	"test_color" "color" "100 100 255 255"
}

Attributes

Heads-up: this list is very incomplete.

DMX files can contain a number of attributes. Every attribute exist in an array form, but the value itself, wether it's in an array or not, is the same size.

Attribute	Type number	Value's size (bytes)	Comment
Element	1	4	The value is an int pointing to an element in the element index.
Int	2	4	Self-explanatory.
Float	3	4	Self-explanatory.
Bool	4	1	The value is a simple byte, which is either 1 or 0.
String	5	N	The value is a null-terminated string of varying size.
Void	6	?	?
Object ID	7	?	?
Color	8	4	The value is 4 bytes, corresponding to RGBA.
Vector 2	9	?	I've not tested this attribute, but the value's size is probably 8 (two floats)
Vector 3	10	12	The value is 3 floats of 4 bytes each.
Vector 4	11	?	I've not tested this attribute, but the value's size is probably 16 (4 floats)
Angle	12	?	I've not tested this attribute, but I suspect it being the same as a Vector 3.
Quaternion	13	?	I've not tested this attribute, but I suspect it being the same as a Vector 4.
Matrix	14	?	I've not tested this attribute, but I remember reading its value's size is 16 floats. Matrices seem to be row-major.

Array counterparts of these attributes possess a different type number. You can simply add 14 to the normal attribute's type to get the array's type. So for example, a string array's type number is 19 (5 + 14).

Random stuff

The structure of the DMX format itself is particular, as in childs "can have multiple parents". This seem to be made possible with the "Element" attribute, which points to elements in the index. Multiple Element attributes can point to the same element.

I have not yet seen the elements' UUIDs having any real purpose in binary form, but this is only based on my experience with the format. I may find one in the future.

Based on my work with tf2pcfm, I've found out that a PCF file, at least concerning TF2, cannot have multiple elements with the same name, even if their UUIDs are different. If it's the case, the game will use only one of the elements throughout the file, disregarding the other ones.