Login Page - Create Account

DTC Protocol Discussion Forum


Date/Time: Fri, 29 Nov 2024 14:39:59 +0000



[Locked] - Python interface for DTC protocol library

View Count: 11292

[2017-04-08 13:16:42]
Queeq - Posts: 42
Hi,

I would like to use DTC protocol from Python, but haven't found any available solutions. Thus, I decided to try SWIG to generate it. As my knowledge about C/C++ is very basic, I encountered some problems during the process. I ended up with the following (doing it on Linux Mint,
GCC 5.4.0, SWIG 3.0.8):

1. Put DTCProtocol.h and DTCProtocol.cpp in a target folder.

2. Create DTCProtocol.i with the following contents:

%module DTCProtocol

%{
#define SWIG_FILE_WITH_INIT
#include "DTCProtocol.h"
%}

%include "DTCProtocol.h"


3. Run the following (according to SWIG manual):
swig -c++ -python DTCProtocol.i
gcc -O2 -fPIC -std=c++11 -c DTCProtocol.cpp
gcc -O2 -fPIC -std=c++11 -c DTCProtocol_wrap.cxx -I/usr/include/python3.5m
gcc -shared DTCProtocol.o DTCProtocol_wrap.o -o _DTCProtocol.so

During the swig run there were many warnings like these:

DTCProtocol.h:15: Warning 315: Nothing known about 'std::int32_t'.
DTCProtocol.h:16: Warning 315: Nothing known about 'std::int64_t'.
DTCProtocol.h:17: Warning 315: Nothing known about 'std::uint16_t'.
DTCProtocol.h:18: Warning 315: Nothing known about 'std::uint32_t'.

Somewhere on Stackoverflow it was mentioned that it is not critical.

Further on the way, I had to add
#include <cfloat>
to DTCProtocol.h, otherwise there were errors like the following:


In file included from DTCProtocol_wrap.cxx:3204:0:
DTCProtocol.h: In member function ‘void DTC::s_MarketDataSnapshot::Clear()’:
DTCProtocol.h:834:29: error: ‘DBL_MAX’ was not declared in this scope
    SessionSettlementPrice = DBL_MAX;
                             ^
DTCProtocol.h: In member function ‘void DTC::s_MarketDataUpdateBidAsk::Clear()’:
DTCProtocol.h:1495:15: error: ‘DBL_MAX’ was not declared in this scope
    BidPrice = DBL_MAX; //This also signifies the BidQuantity is unset
               ^
DTCProtocol.h: In member function ‘void DTC::s_MarketDataUpdateBidAskCompact::Clear()’:
DTCProtocol.h:1574:15: error: ‘FLT_MAX’ was not declared in this scope
    BidPrice = FLT_MAX;
               ^
DTCProtocol.h: In member function ‘void DTC::s_OrderUpdate::Clear()’:
DTCProtocol.h:2493:13: error: ‘DBL_MAX’ was not declared in this scope
    Price1 = DBL_MAX;
             ^

When trying to import the resulting object, it gives the following errors:


➜ python3
Python 3.5.2 (default, Nov 17 2016, 17:05:23)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import DTCProtocol
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/queeq/Work/wat/wat/DTC/DTCProtocol.py", line 28, in <module>
_DTCProtocol = swig_import_helper()
File "/home/queeq/Work/wat/wat/DTC/DTCProtocol.py", line 24, in swig_import_helper
_mod = imp.load_module('_DTCProtocol', fp, pathname, description)
File "/usr/lib/python3.5/imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "/usr/lib/python3.5/imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: /home/queeq/Work/wat/wat/DTC/_DTCProtocol.so: undefined symbol: _ZN3DTC25s_MarketDepthFullUpdate2016NUM_DEPTH_LEVELSE

I tried readelf on the .so file, and it seems that the above function (or whatever it is) is there:

➜ readelf -sW _DTCProtocol.so | grep _ZN3DTC25s_MarketDepthFullUpdate2016NUM_DEPTH_LEVELSE
29: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND _ZN3DTC25s_MarketDepthFullUpdate2016NUM_DEPTH_LEVELSE
3407: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND _ZN3DTC25s_MarketDepthFullUpdate2016NUM_DEPTH_LEVELSE

I don't really know how should it look like. Could someone help me with this? Would be really great to have DTC protocol available for use in Python, and with SWIG it seems to be pretty straightforward.

Thanks!

P.S. SWIG manual used is here: http://www.swig.org/Doc3.0/Python.html#Python
Date Time Of Last Edit: 2017-04-08 19:24:57
[2017-04-09 16:29:17]
Queeq - Posts: 42
I finally managed to make it importable. Had to add some ignores to DTCProtocol.i:

%module DTCProtocol

%{
#define SWIG_FILE_WITH_INIT
#include "DTCProtocol.h"
%}

%ignore s_MarketDepthFullUpdate20;
%ignore s_MarketDepthFullUpdate10;

%include "DTCProtocol.h"

And also add -lstdc++ when creating .so. Thus, the complete list of commands is as follows:

swig -c++ -python DTCProtocol.i
gcc -O2 -fPIC -std=c++11 -c DTCProtocol.cpp DTCProtocol_wrap.cxx -I/usr/include/python3.5
gcc -shared DTCProtocol.o DTCProtocol_wrap.o -o _DTCProtocol.so -lstdc++

Seems to work now:

>>> import DTCProtocol
>>> print(DTCProtocol.cvar.CURRENT_VERSION)
<Swig Object of type 'int32_t *' at 0x7fc50173fa20>

Not sure how to use it from a Python script yet, but I think I'll figure it out.
Date Time Of Last Edit: 2017-04-09 16:31:40
[2017-04-09 17:06:32]
Queeq - Posts: 42
Alright, had to include stdint.i to DTCProtocol.i, so the whole file now looks like this:

%module DTCProtocol

%include "stdint.i"

%{
#define SWIG_FILE_WITH_INIT
#include "DTCProtocol.h"
%}

%ignore s_MarketDepthFullUpdate20;
%ignore s_MarketDepthFullUpdate10;

%include "DTCProtocol.h"

Now:

>>> import DTCProtocol
>>> print(DTCProtocol.CURRENT_VERSION)
7

[2017-04-10 17:41:42]
DTC Engineering - Posts: 320
In regards to Python, we recommend using Google Protocol Buffers encoding instead.

Here is the Google documentation for that:

https://developers.google.com/protocol-buffers/docs/pythontutorial

The current protocol buffer file can be found here:
http://dtcprotocol.org/DTC_Files/DTCProtocol.proto
Date Time Of Last Edit: 2017-04-10 17:42:29
[2017-04-17 07:53:32]
Queeq - Posts: 42
Thanks for the hint. I tried to test it, but I can't make it work.

What I did is I compiled the proto3 file, taken from my SC installation, using latest google.protobuf. It compiled successfully and I am able to access messages from Python. I wrote the following script as a test:


""" Data retrieval from SierraChart services """

import socket
import struct

from DTC import DTCProtocol_pb2 as Dtc


def construct_message(m, message_type):
""" Prepends message size and type, appends ProtocolType and packs it to binary

Args:
m (DTCProtocol): DTC protobuf message
message_type (int): DTC message type

Returns:
bytes: Binary string
"""
try:
# Try to set ProtocolVersion as most messages contain this field
m.ProtocolVersion = Dtc.CURRENT_VERSION
except AttributeError:
pass
m.ProtocolType = "DTC" # ProtocolType is always the same
total_len = 4 + m.ByteSize() # 2 bytes Size + 2 bytes Type
header = struct.pack('<HH', total_len, message_type) # Prepare 4-byte little-endian header
binary_message = m.SerializeToString()
print(binary_message) # <-- This print outputs b'\x08\x07\x1a\x03DTC', which is wrong
return header + binary_message

SC_INTRADAY = ("127.0.0.1", 11100)
SC_HISTORICAL = ("127.0.0.1", 11101)
ENCODING = Dtc.BINARY_ENCODING

# Create encoding request
message = Dtc.EncodingRequest()
message.Encoding = ENCODING
data = construct_message(message, Dtc.ENCODING_REQUEST)

# Connect
s = socket.socket()
s.connect(SC_INTRADAY)

# Send encoding request
s.send(data)
# Receive encoding response
response = s.recv(1024)
# 2 bytes with offset of 2 must indicate this is encoding response message type
assert struct.unpack_from('<H', response, 2)[0] == Dtc.ENCODING_RESPONSE
encoding_response = Dtc.EncodingResponse()
print(encoding_response.ParseFromString(response[4:]).Encoding)

For some reason, EncodingRequest message doesn't get correctly serialized: it gives b'\x08\x07\x1a\x03DTC'. As a result, SC logs "Unexpected DTC encoding request: 4412484".

I'm pretty stuck here and would appreciate some help.

Thanks!
[2017-04-17 17:43:30]
Queeq - Posts: 42
I figured I did it wrong: would need to set/negotiate the protobuf encoding on/with the server.
[2017-04-17 21:46:16]
DTC Engineering - Posts: 320
Yes, the client makes a request to the server for the encoding.

Also we recommend using this DTC protocol buffer definition file instead of version 3 :

http://dtcprotocol.org/DTC_Files/DTCProtocol.proto
Date Time Of Last Edit: 2017-04-17 22:02:04
[2017-04-17 21:58:10]
Queeq - Posts: 42
Why is it recommended? I'm kind of used to the notion that "latest=best" :)
[2017-04-17 22:01:05]
DTC Engineering - Posts: 320
It has not been tested by us and may not have all of the latest updates.
Date Time Of Last Edit: 2017-04-17 22:01:34
[2020-02-16 12:26:30]
User824913 - Posts: 15
Hi,
can someone help me, please?
I don't undersant: " figured I did it wrong: would need to set/negotiate the protobuf encoding on/with the server."

thank you
[2020-02-16 19:10:50]
Queeq - Posts: 42
Maybe looking at the code could help you: https://github.com/Queeq/pydtc
Or just use the library instead of reinventing it all over again.
[2020-12-23 08:01:21]
RoadKill - Posts: 38
@Queeq
It's not entirely clear what you say:
would need to set/negotiate the protobuf encoding on/with the server

0. Where is the info for this?
1. What was wrong with the example code above?
2. The example code from above is not in your repo.
3. Should we understand that in your repo, you are using the procolbuffer library, whereas the "example" above is raw parsing?
[2020-12-23 09:23:06]
Queeq - Posts: 42
Sigh, that message is almost 5 years old. I'll try to answer though.

0. I think I deduced this myself
1. I don't remember. It's obsolete, look at the code in repo, it works.
2. The program has evolved since then.
3. The repo does use protobuf indeed. The example above I don't remember and I don't want to try to analyze it because it's irrelevant.
[2020-12-23 10:02:55]
RoadKill - Posts: 38
I see. The problem with what's in your repo is that there are no informative examples.

It would be great to see how you select and place some of the basic DTC commands,
using your library. For example:

1. How to get live tick price data (MARKET_DATA_UPDATE_BID_ASK)
2. How to get order book depth (MARKET_DEPTH_UPDATE_LEVEL)
[2020-12-23 11:18:52]
Queeq - Posts: 42
There's no support for fetching order data. If you'd like to add it - merge request is welcome. In general it may be reasonable to start at the function that is used to compose messages for requesting historical data (DtcConnection.historical_data_request): https://github.com/Queeq/pydtc/blob/18da88f388712d398990a215fb093f82268bd812/pydtc/main.py#L382

I vaguely remember that I decided not to implement it because it would've required quite a lot of work as historical data request is way simpler: it's just one request per connection. Other types of data, e.g. live feed is more complicated. Maybe order data would be as simple as historical one, but then it would also require some mechanism to reuse existing connection, i.e. pass DtcConnection object around. With some effort anyone can do this - there should be enough docstrings and comments in the code to understand how the library works.

To post a message in this thread, you need to log in with your Sierra Chart account:

Login

Login Page - Create Account