Xmlpktgen

Description

XML Packet Generator, abbreviated xmlpktgen, is a tool specifically designed to solve one of the problems with Python interaction: its dynamic typing.

Using marshaling and unmarshaling, we are able to reproduce any object that a client sends to us; however, there is nothing we can say about these objects at compile time: are these ints? floats? tuples? Thus the objects passed to calls as arguments (for example) must always be examined and their types must be determined in order to be effectively used in C++.

We could do this manually; however, it's soon clear that it's nothing but endless code repetition. So we use the xmlpktgen: if we feed it with proper XML destribing the object(s) we wish to decode, it generates .h/.cpp files with classes that do exactly what we need - scan through given object and pull out stuff we need (ints, strings etc.), according to the given XML description. These classes can also handle inverse operation: given the components, they can construct quite complex objects out of them.

Current usage

In the current setup, its headers are in include/xmlpktgen, source is in src/xmlpktgen. After it's built, it's used to generate .h/.cpp files for .xmlp descriptors in src/eve-common/packets. These generated classes are then used throughout the whole eve-common, eve-tool and eve-server source.

Example on a real packet

The packet is to big to be pasted here so you can access it here: TBD

Let's take this complicated packet as an example on how to create the xmlp code in these situations.

First, we look at what we have as a general structure. You start by looking under the [PySubstream] line. There we have a PyString called "CreateCharacterWithDoll" - this is the method that has been called from the client to the server. The client sends information about your character creation results (name, race, gender, bloodline, avatar appearance, picture pose information and schoolID).

Under the PyString "CreateCharacterWithDoll" there is a tuple containing 7 items. If you take a close look you can see that the first item it's a string that contains the name of the created character and the rest of the PyInts and PyObjects are the data specified above.

The firs thing you want to do is to add a <elementDef name="CallCreateCharacterWithDoll">, this will define a class named CallCreateCharacterWithDoll that you can later use inside the C++ code. Then you take the first thing under the PySubstream, which is a <tupleInline> and under this you add the elements one by one like this:

<elementDef name="CallCreateCharacterWithDoll">
 <tupleInline>
  <wstring name = "name" />
  <int name = "bloodlineID" />
  <int name = "ancestryID" />
  <objectInline>
   <stringInline value = "util.KeyVal" />
   <dict name = "avatarInfo" />
  </objectInline>
  <objectInline>
   <stringInline value = "util.KeyVal" />
   <dict name = "poseInfo" />
  </objectInline>
  <int name = "schoolID" />
 </tupleInline>
</elementDef> 

The name after each type (wstring, int, list) specifies the name of the property that will be generated inside the class CallCreateaCharacterWithDoll, it's like saying "int number" in ordinary C++. First 3 lines were pretty straight forward, a string and 2 ints, but what about the fourth one? Well... here is the tricky part... This is an object which consists of a string and a dict; the string "util.KeyVal" describes the dict format which is: "Key Val" (ex: "gold":99999) a key, named "gold" for example and it's value 99999 which can be an int, float, list or any object.

I've chosen this packet because of it's complexity from which you can learn how to deal with types other than the basic int, float, string.

Coming soon: The continuation of the the process on how we go about extracting the data and store it in the Data Base.

XMLP Format Demystified

XMLP Structure Using it on the C++ side:

<objectInline>
  <stringInline value="util.KeyVal" />
  <dictInline>
    <dictInlineEntry key="itemID">
      <int name="itemID" />
    </dictInlineEntry>
    <dictInlineEntry key="attributes">
      <dictInt name="attributes" />
    </dictInlineEntry>
    <dictInlineEntry key="invItem">
      <raw name="invItem" />                            obj.invItem = new PyPackedRow(...)
    </dictInlineEntry>
    <dictInlineEntry key="time">
    </dictInlineEntry>
      <int name="time" />
    <dictInlineEntry key="activeEffects">
      <dictInt name="activeEffects" />
    </dictInlineEntry>
  </dictInline>
</objectInline>

example for including an entire xmlp structure into another:

<objectInline>
  <stringInline value=util.KeyVal" />
  <dictInline>
    <dictInlineEntry key="itemID">
      <element name="notes" type="util_Rowset" />
    </dictInlineEntry>
  </dictInline>
</objectInline>

Tags and their uses

A full list of xmlp tags can be found in Generator.h/.cpp in the "xmlpktgen" project. Use <> not [] wiki formating is evil.

[elementDef] [/elementDef]

the outermost wrapper tag around an entire XMLP structure, close with

[element name="notes" type="util_Rowset" /]

you supply the actual object (not just a pointer) to the util_Rowset

[elementPtr type="OnLSC_SenderInfo" name="sender" /]

you supply a pointer to the OnLSC_SenderInfo class object instance

[raw name="raw_name" /]

you put a new PyRep * object here, which can be any PyXXXX class pointer

[objectEx name="name" type="a type" optional="true/false"] [/objectEx]

specifies a PyObjectEx object, Type it's like util.KeyVal object type and can be left out, Optional asks if it's type 2 or not

[objectInline]

specifies a PyObject object, closed by [/objectInline]

[substreamInline] marshalling/unmarshalling [/substreamInline]

specifies a PySubStream object - this is used by lower level

[none /]

specifies a PyNone object

[dictInline] [/dictInline]

wrap a dictionary in this and its closing tag

[dictInlineEntry key="dict_key_string"]

this defines an expected dictionary key value as a string "dict_key_string", used inside [dictinline] you then need to specify on the next line a value part of this dictionary key-value pair eg:

<dictInlineEntry key="key_string">
   <int name="int_name" />
</dictInlineEntry>

[dict name="dict_name" /]

defines a whole dictionary object of type PyDict

[dictInt name="dict_int_name" /]

defines a whole dictionary with integers used for ALL key values

[dictStr name="dict_string_name" /]

defines a whole dictionary with strings used for ALL key values

[dictRaw name="dict_raw_name" key="key_Ctype" pykey="key_Ptype" value="val_Ctype" pyvalue="val_Ptype" /]

defines a whole dictionary with "key_Ctype" of a C/C++ type such as uint32 as keys, with the Python type specified type specified by "key_Ptype", the value C/C++ type specified by "val_Ctype", and the Python type specified by "val_Ptype"

[stringInline value="string_value" /]

specifies a constant PyString value that contains the string "string_value"

[tokenInline]

specifies a PyToken

[tupleInline]

wrap a tuple (list of items) in this and its closing tag </tupleInline>

[tuple name="tuple_name" /]

specifies a tuple object of type PyTuple

[listInline]

wrap a list (list of items) in this and its closing tag </listInline>

[list name="list_name" /]

specifies a list of objects with the name "list_name"

[listInt name="list_int_name" /]

specifies a list of PyInt objects with the name "list_int_name"

[listStr name="list_string_name" /]

specifies a list of PyString objects with the name "list_string_name"; no explicit

[string name="string_name" /]

simple object type PyString with the name "string_name"

[wstring name="wstring_name" /]

simple object type PyWString with the name "wstring_name"

[real name="real_name" /]

simple object type PyFloat with the name "real_name"

[int name="int_name" /]

simple object type PyInt with the name "int_name"

[long name="long_name" /]

simple object type PyLong with the name "long_name"

[bool name="bool_name" /]

simple object type PyBool with the name "bool_name"