Support Board
Date/Time: Sat, 23 Nov 2024 18:09:08 +0000
Post From: Python for Sierra Chart
[2021-06-18 18:10:52] |
biconoid - Posts: 5 |
Trying to read Sierra Chart SCID file using Python and export as CSV/TSV Using following approach: With reference to s_IntradayRecord Structure Member Descriptions at Intraday Data File Format - The Intraday data record structure, s_IntradayRecord, is 40 bytes in size. - The Intraday data file header, s_IntradayHeader, is 56 bytes in size. - The DateTime member variable is a SCDateTimeMS variable. This is a 64-bit integer. It represents the number of microseconds since the SCDateTime epoch of December 30, 1899. s_IntradayRecord()// Constructor
struct s_IntradayRecord { static const float SINGLE_TRADE_WITH_BID_ASK; static const float FIRST_SUB_TRADE_OF_UNBUNDLED_TRADE; static const float LAST_SUB_TRADE_OF_UNBUNDLED_TRADE; s_IntradayRecord()// Constructor PYTHON PART Sierra Chart | { Type | https://docs.python.org/3/library/struct.html ----------------------- |---------------------------------- | ----------------------------------------------- SCDateTimeMS DateTime; | DateTime; d | float Open; | Open = 0.0; f | Format C Python type Standard float High; | High = 0.0; f | Type Type Size(bytes) float Low; | Low = 0.0; f | ------------------------------------------ float Close; | Close = 0.0; f | f float float 4 u_int32 NumTrades; | NumTrades = 0; I | I unsigned int integer 4 u_int32 TotalVolume; | TotalVolume = 0; I | d double float 8 u_int32 BidVolume; | BidVolume = 0; I | u_int32 AskVolume; | AskVolume = 0; I | | } }; Procedure: Data unpacking as: size_of_header = 0x36 (hex) # 56 bytes # Discarded
size_of_record = 0x28 (hex) # 40 bytes (<d 4f 4I) # Refer to above table unpacked_data = struct.unpack('<d 4f 4I', data) # Further exported and saved to files like below # Following TSV (CSV) file is obtainted after getting the data.scid (check attached) $ cat data.tsv Date Open High Low Close Volume NumberOfTrades BidVolume AskVolume 1.8752101406881474e-308 2623.0 0.0 2623.0 2623.0 1 1 1 0 1.8754475234208735e-308 2717.0 0.0 2717.0 2717.0 1 1 1 0 1.875547350372747e-308 2750.0 0.0 2750.0 2750.0 1 1 1 0 1.8757877222026304e-308 2742.5 2757.75 2742.5 2742.5 1 1 1 0 1.876050741013588e-308 2830.0 2830.0 2805.75 2830.0 5 0 0 5 1.876054622887367e-308 2845.0 2845.0 2814.75 2845.0 3 0 0 3 1.876054699961608e-308 2845.0 2845.0 2814.75 2845.0 2 0 0 2 1.8761342153807153e-308 2900.0 2900.0 2870.75 2900.0 3 0 0 3 1.876134249471245e-308 2900.0 2900.0 2870.75 2900.0 2 0 0 2 1.876686289274035e-308 2890.0 2918.25 2890.0 2890.0 1 1 1 0 $ python3
Note: Above dataframe output is almost same EXCEPT DATETIME as if i export SCID to a text file and import into a dataframe.Python 3.8.5 (default, May 27 2021, 13:30:53) [GCC 9.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pandas as pd >>> pd.read_csv('data.tsv', sep='\t') Date Open High Low Close Volume NumberOfTrades BidVolume AskVolume 0 1.875210e-308 2623.0 0.00 2623.00 2623.0 1 1 1 0 1 1.875448e-308 2717.0 0.00 2717.00 2717.0 1 1 1 0 2 1.875547e-308 2750.0 0.00 2750.00 2750.0 1 1 1 0 3 1.875788e-308 2742.5 2757.75 2742.50 2742.5 1 1 1 0 4 1.876051e-308 2830.0 2830.00 2805.75 2830.0 5 0 0 5 5 1.876055e-308 2845.0 2845.00 2814.75 2845.0 3 0 0 3 6 1.876055e-308 2845.0 2845.00 2814.75 2845.0 2 0 0 2 7 1.876134e-308 2900.0 2900.00 2870.75 2900.0 3 0 0 3 8 1.876134e-308 2900.0 2900.00 2870.75 2900.0 2 0 0 2 9 1.876686e-308 2890.0 2918.25 2890.00 2890.0 1 1 1 0 >>> Manual conversion of SCID file to TXT via SierraChart #ESM21-CME-Sample.txt Date, Time, Open, High, Low, Last, Volume, NumberOfTrades, BidVolume, AskVolume
2020/4/8, 00:32:59, 2623.00, 0.00, 2623.00, 2623.00, 1, 1, 1, 0 2020/4/13, 14:00:47, 2717.00, 0.00, 2717.00, 2717.00, 1, 1, 1, 0 2020/4/15, 22:08:19, 2750.00, 0.00, 2750.00, 2750.00, 1, 1, 1, 0 2020/4/21, 13:16:57, 2742.50, 2757.75, 2742.50, 2742.50, 1, 1, 1, 0 2020/4/27, 17:09:33, 2830.00, 2830.00, 2805.75, 2830.00, 5, 1, 0, 5 2020/4/27, 19:20:30, 2845.00, 2845.00, 2814.75, 2845.00, 3, 1, 0, 3 2020/4/27, 19:23:06, 2845.00, 2845.00, 2814.75, 2845.00, 2, 1, 0, 2 2020/4/29, 16:05:27, 2900.00, 2900.00, 2870.75, 2900.00, 3, 1, 0, 3 ---------------------------------------------------------------------------------------------------------------- Issue: ---------------------------------------------------------------------------------------------------------------- Unable to extract/convert the datetime part to a proper value. Anyone has any pointers. Tried via - numpy - datetime module Sample Python code ----------------- $ python3 Python 3.8.5 (default, May 27 2021, 13:30:53) [GCC 9.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from datetime import datetime >>> datetime.fromtimestamp(1.8752101406881474e-308) datetime.datetime(1969, 12, 31, 19, 0) >>> FAIL: That's an error Sample Python code ----------------- import numpy as np
FAIL: That's an errorfrom time import time """ From: Intraday Data File Format The DateTime member variable is a SCDateTimeMS variable. This is a 64-bit integer. It represents the number of microseconds since the SCDateTime epoch of December 30, 1899. """ # Single parameters ep = np.datetime64('1899-12-30') # epoch date tz = np.timedelta64(5, 'h')/np.timedelta64(1, 'D') # time-zone adjustment UTC +5 dv = 1.8752101406881474e-308 # d[0] value from struct.unpack('<d 4f 4I') refer to TSV file or DataFrame above dt = dv + tz # adjusted time # Try to get full date ms = (np.timedelta64(int(round((dt-int(dt))*86400)),'ms')) # Delta in millisecond for current day my_time = ep + np.timedelta64(int(dt)) + ms # Epoch Time + Delta + millisecond spent print(my_time) #Output ------------------------- 1899-12-30T00:00:18.000 # Attachment # data.scid - is a subset of 'ESM21-CME.scid' 2.5Gb file i've only extracted 456 bytes for simplifying data extraction (56 + 400 bytes) 56 bytes = header 400 bytes = 40 x 10 intraday records |
data.scid - Attached On 2021-06-18 18:09:39 UTC - Size: 457 B - 653 views |