Support Board
Date/Time: Sat, 23 Nov 2024 15:32:59 +0000
[User Discussion] - Python for Sierra Chart
View Count: 38864
| [2019-06-27 15:33:37] |
| User183724 - Posts: 183 |
|
Kiwi Quick question...do you have a SCID to TEXT version of your Text2SCid.py that you would be willing to share? I'd like to read the SCID file into a program as a text or CSV file. Im new to Python (used a lot of other languages)... have been using SC for a while...I kind of understand what youre doing here and would be interesting in trying to develop something. Thank You |
| [2019-06-27 22:39:17] |
| Kiwi - Posts: 375 |
|
This will probably work. It converts an scid file (named after the imports) to a tab separated variable file of the same name (but .scid becomes .tsv) & path. Let me know if you have problems (including any error messages). It's updated for Python 3 & tested with 3.7.4. Date Time Of Last Edit: 2019-06-27 22:39:57
|
 SCID_to_TSV.py - Attached On 2019-06-27 22:38:54 UTC - Size: 3.55 KB - 880 views |
| [2019-06-28 00:23:47] |
| User183724 - Posts: 183 |
|
Kiwi- Thanks for the file. I'll let you know how it goes
|
| [2019-06-28 16:28:36] |
| User183724 - Posts: 183 |
|
Has anyone worked with the SCSS spreadsheet files? Do you have py files that read/write to these files that you would be willing to share with me? Thanks in advance
|
| [2019-07-01 20:57:55] |
| |
|
The Spreadsheet file format is too complex to be working with in this way.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2019-07-01 20:58:08
|
| [2019-07-02 00:49:00] |
| User183724 - Posts: 183 |
|
Thanks for the reply. I'm really just looking for options on how to read real time data into Python. Spread sheet SCSS files were just something I was looking at
|
| [2019-09-16 15:29:03] |
| Mack - Posts: 83 |
|
What is really missing is a Python framework for SierraChart. Something that allows to code, backtest and review strategies. @Kiwi: in your opinion, would it be better to develop a standalone backtest system? or something that is tightly integrated with SierraChart? In theory, you could just use SC to get data, and execute signals. (but do the backtesting in your own backtester) Date Time Of Last Edit: 2019-09-16 15:31:58
|
| [2021-06-18 18:10:52] |
| biconoid - Posts: 5 |
|
Trying to read Sierra Chart SCID file using Python and export as CSV/TSV Using following approach: With reference to s_IntradayRecord Structure Member Descriptions at Intraday Data File Format - The Intraday data record structure, s_IntradayRecord, is 40 bytes in size. - The Intraday data file header, s_IntradayHeader, is 56 bytes in size. - The DateTime member variable is a SCDateTimeMS variable. This is a 64-bit integer. It represents the number of microseconds since the SCDateTime epoch of December 30, 1899. s_IntradayRecord()// Constructor
struct s_IntradayRecord { static const float SINGLE_TRADE_WITH_BID_ASK; static const float FIRST_SUB_TRADE_OF_UNBUNDLED_TRADE; static const float LAST_SUB_TRADE_OF_UNBUNDLED_TRADE; s_IntradayRecord()// Constructor PYTHON PART Sierra Chart | { Type | https://docs.python.org/3/library/struct.html ----------------------- |---------------------------------- | ----------------------------------------------- SCDateTimeMS DateTime; | DateTime; d | float Open; | Open = 0.0; f | Format C Python type Standard float High; | High = 0.0; f | Type Type Size(bytes) float Low; | Low = 0.0; f | ------------------------------------------ float Close; | Close = 0.0; f | f float float 4 u_int32 NumTrades; | NumTrades = 0; I | I unsigned int integer 4 u_int32 TotalVolume; | TotalVolume = 0; I | d double float 8 u_int32 BidVolume; | BidVolume = 0; I | u_int32 AskVolume; | AskVolume = 0; I | | } }; Procedure: Data unpacking as: size_of_header = 0x36 (hex) # 56 bytes # Discarded
size_of_record = 0x28 (hex) # 40 bytes (<d 4f 4I) # Refer to above table unpacked_data = struct.unpack('<d 4f 4I', data) # Further exported and saved to files like below # Following TSV (CSV) file is obtainted after getting the data.scid (check attached) $ cat data.tsv Date Open High Low Close Volume NumberOfTrades BidVolume AskVolume 1.8752101406881474e-308 2623.0 0.0 2623.0 2623.0 1 1 1 0 1.8754475234208735e-308 2717.0 0.0 2717.0 2717.0 1 1 1 0 1.875547350372747e-308 2750.0 0.0 2750.0 2750.0 1 1 1 0 1.8757877222026304e-308 2742.5 2757.75 2742.5 2742.5 1 1 1 0 1.876050741013588e-308 2830.0 2830.0 2805.75 2830.0 5 0 0 5 1.876054622887367e-308 2845.0 2845.0 2814.75 2845.0 3 0 0 3 1.876054699961608e-308 2845.0 2845.0 2814.75 2845.0 2 0 0 2 1.8761342153807153e-308 2900.0 2900.0 2870.75 2900.0 3 0 0 3 1.876134249471245e-308 2900.0 2900.0 2870.75 2900.0 2 0 0 2 1.876686289274035e-308 2890.0 2918.25 2890.0 2890.0 1 1 1 0 $ python3
Note: Above dataframe output is almost same EXCEPT DATETIME as if i export SCID to a text file and import into a dataframe.Python 3.8.5 (default, May 27 2021, 13:30:53) [GCC 9.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pandas as pd >>> pd.read_csv('data.tsv', sep='\t') Date Open High Low Close Volume NumberOfTrades BidVolume AskVolume 0 1.875210e-308 2623.0 0.00 2623.00 2623.0 1 1 1 0 1 1.875448e-308 2717.0 0.00 2717.00 2717.0 1 1 1 0 2 1.875547e-308 2750.0 0.00 2750.00 2750.0 1 1 1 0 3 1.875788e-308 2742.5 2757.75 2742.50 2742.5 1 1 1 0 4 1.876051e-308 2830.0 2830.00 2805.75 2830.0 5 0 0 5 5 1.876055e-308 2845.0 2845.00 2814.75 2845.0 3 0 0 3 6 1.876055e-308 2845.0 2845.00 2814.75 2845.0 2 0 0 2 7 1.876134e-308 2900.0 2900.00 2870.75 2900.0 3 0 0 3 8 1.876134e-308 2900.0 2900.00 2870.75 2900.0 2 0 0 2 9 1.876686e-308 2890.0 2918.25 2890.00 2890.0 1 1 1 0 >>> Manual conversion of SCID file to TXT via SierraChart #ESM21-CME-Sample.txt Date, Time, Open, High, Low, Last, Volume, NumberOfTrades, BidVolume, AskVolume
2020/4/8, 00:32:59, 2623.00, 0.00, 2623.00, 2623.00, 1, 1, 1, 0 2020/4/13, 14:00:47, 2717.00, 0.00, 2717.00, 2717.00, 1, 1, 1, 0 2020/4/15, 22:08:19, 2750.00, 0.00, 2750.00, 2750.00, 1, 1, 1, 0 2020/4/21, 13:16:57, 2742.50, 2757.75, 2742.50, 2742.50, 1, 1, 1, 0 2020/4/27, 17:09:33, 2830.00, 2830.00, 2805.75, 2830.00, 5, 1, 0, 5 2020/4/27, 19:20:30, 2845.00, 2845.00, 2814.75, 2845.00, 3, 1, 0, 3 2020/4/27, 19:23:06, 2845.00, 2845.00, 2814.75, 2845.00, 2, 1, 0, 2 2020/4/29, 16:05:27, 2900.00, 2900.00, 2870.75, 2900.00, 3, 1, 0, 3 ---------------------------------------------------------------------------------------------------------------- Issue: ---------------------------------------------------------------------------------------------------------------- Unable to extract/convert the datetime part to a proper value. Anyone has any pointers. Tried via - numpy - datetime module Sample Python code ----------------- $ python3 Python 3.8.5 (default, May 27 2021, 13:30:53) [GCC 9.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from datetime import datetime >>> datetime.fromtimestamp(1.8752101406881474e-308) datetime.datetime(1969, 12, 31, 19, 0) >>> FAIL: That's an error Sample Python code ----------------- import numpy as np
FAIL: That's an errorfrom time import time """ From: Intraday Data File Format The DateTime member variable is a SCDateTimeMS variable. This is a 64-bit integer. It represents the number of microseconds since the SCDateTime epoch of December 30, 1899. """ # Single parameters ep = np.datetime64('1899-12-30') # epoch date tz = np.timedelta64(5, 'h')/np.timedelta64(1, 'D') # time-zone adjustment UTC +5 dv = 1.8752101406881474e-308 # d[0] value from struct.unpack('<d 4f 4I') refer to TSV file or DataFrame above dt = dv + tz # adjusted time # Try to get full date ms = (np.timedelta64(int(round((dt-int(dt))*86400)),'ms')) # Delta in millisecond for current day my_time = ep + np.timedelta64(int(dt)) + ms # Epoch Time + Delta + millisecond spent print(my_time) #Output ------------------------- 1899-12-30T00:00:18.000 # Attachment # data.scid - is a subset of 'ESM21-CME.scid' 2.5Gb file i've only extracted 456 bytes for simplifying data extraction (56 + 400 bytes) 56 bytes = header 400 bytes = 40 x 10 intraday records |
| data.scid - Attached On 2021-06-18 18:09:39 UTC - Size: 457 B - 653 views |
| [2021-06-18 19:51:38] |
| User907968 - Posts: 823 |
|
Your unpack is not correct, you are using double for the date time variable, which it is not.
|
| [2021-06-18 20:09:53] |
| biconoid - Posts: 5 |
|
As per https://docs.python.org/3/library/struct.html there are only three options to unpack 64 bits (8 bytes), i.e. q, Q and d. What would be correct structure to unpack? Format C Python Standard
Type Type Size(bytes) --------------------------------------------------- Q unsigned long long integer 8 q long long integer 8 d double float 8 Also Sierra documentation mentions Versions prior to 2151, use a double precision floating point type variable to represent Date-Time values.
Just for testing purposes i've tested using q and Q. $ cat files/data.tsv
Date Open High Low Close Volume NumberOfTrades BidVolume AskVolume 3795467579000000.0 2623.0 0.0 2623.0 2623.0 1 1 1 0 3795948047000000.0 2717.0 0.0 2717.0 2717.0 1 1 1 0 3796150099000000.0 2750.0 0.0 2750.0 2750.0 1 1 1 0 3796636617000000.0 2742.5 2757.75 2742.5 2742.5 1 1 1 0 When unpacking using Q or q. >>> datetime.fromtimestamp(3795467579)
datetime.datetime(2090, 4, 9, 20, 32, 59) >>> and >>> datetime.fromtimestamp(3795467579000000.0 )
Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: year 120275549 == out of range >>> Date Time Of Last Edit: 2021-06-18 20:26:19
|
| [2021-06-18 22:20:48] |
| Kiwi - Posts: 375 |
|
Here's a python file that converts an scid to csv with defaults for eur.usd in my data directory. Feel free to extract what you need or modify and reuse. """
Takes a SierraChart scid file (input argument 1) and converts it to a comma separated file (comma) Timezone conversion can follow the users local timezone, or a specified integer (input l or an integer but if the default filename is being used, '' must be specified for the filename) Inputs: filename & tz adjust (else UTC). if the filename is 'all' then all scids in directory are converted """ import csv import re import sys import os import struct from datetime import date, datetime, time, timedelta def deserialize(excelDateAndTime): """ Returns excel date time as Python datetime object """ date_tok = int(excelDateAndTime) time_tok = round(86400*(excelDateAndTime - date_tok)) d = date(1899, 12, 30) + timedelta(date_tok) t = time(*helper(time_tok, (24, 60, 60, 1000000))) return datetime.combine(d, t) def helper(factor, units): factor /= 86399.99 result = list() for unit in units: value, factor = divmod(factor * unit, 1) result.append(int(value)) result[3] = int(0) return result def datetimeTwo(excelDateAndTime, tzAdjust): """ Returns excel date time in format YYYYMMDD HHMMSS """ t = str(deserialize(excelDateAndTime + tzAdjust.seconds/86400.0)) return (re.sub(r'(:|-)', '', t)[:8], re.sub(r'(:|-)', '', t)[9:15]) def excelTimeAdjust(date_and_time, tzAdjust): return date_and_time + tzAdjust.seconds/86400.0 def convertRecords(filename, fileoutput, tzvar, dtformat): """ Read in records from SierraChart .scid data filename dtformat determines the datetime representation in column 1 (and 2) """ sizeHeader = 0x38 sizeRecord = 0x28 if tzvar == 99999: tzAdjust = datetime.fromtimestamp(0) - datetime.utcfromtimestamp(0) else: tzAdjust = timedelta(hours=tzvar) header = ('Date', 'Time', 'O', 'H', 'L', 'C', 'V', 'T') ftsv = open(fileoutput, 'w') fileout = csv.writer(ftsv, delimiter=',') with open(filename, 'rb') as fscid: fscid.read(sizeHeader) # discard header fileout.writerow(header) data = fscid.read(sizeRecord) while data: dataRow = struct.unpack('d4f4I', data) if dtformat == 0: adjustedTime = dataRow[0] + tzAdjust.seconds/86400.0 outrow = (str(adjustedTime), str(adjustedTime), str(dataRow[1]), str(dataRow[2]), str(dataRow[3]), str(dataRow[4]), str(dataRow[5]), str(dataRow[6])) fileout.writerow(outrow) elif dtformat == 1: date, time = datetimeTwo(dataRow[0], tzAdjust) outrow = (date, time, str(dataRow[1]), str(dataRow[2]), str(dataRow[3]), str(dataRow[4]), str(dataRow[5]), str(dataRow[6])) fileout.writerow(outrow) data = fscid.read(sizeRecord) return if __name__ == '__main__': filename = '/home/john/zRamdisk/Data/EUR.USD-CASH-IDEALPRO.scid' tzvar = 0 if len(sys.argv) > 1: filename = sys.argv[1] if len(sys.argv) > 2: print(sys.argv[2], type(sys.argv[2])) if sys.argv[2] == 'l': tzvar = 99999 print(tzvar, type(tzvar)) else: tzvar = float(sys.argv[2]) print(tzvar, type(tzvar)) # filename = 'all' ############## remove ########################## if filename != "all": convertRecords(filename, filename[:-4] + 'csv', tzvar, 1) else: files = os.listdir(path='.') for filename in files: if filename[-5:] == '.scid': print("Converting this file", filename, "using Tz", tzvar) convertRecords(filename, filename[:-4] + 'csv', tzvar, 1) |
| [2021-06-19 06:29:49] |
| biconoid - Posts: 5 |
|
Hi Kiwi, Thank you so much for your code. It was well written and consie. Upon running, i'm still seeing the same issue of - datatime reported from 1899, SCDateTime epoch of December 30, 1899 in column[0] and - timestamp as 000000 in column[1] So i've a few questions going fwd. - eur.usd data you're mentioning, it is spot FX market? - Time based or tick based? - If time based, what's the time series difference? - Have you ran this with futures data? In my case, i have only tried to run on Futures/FX obtained by Denali feed - 1 tick ES (SC v2232) - 10 tick GC (SC v2275) - 10 seconds EURUSD) (SC v2275) In all above cases i get wrong datetime value. Output Date,Time,O,H,L,C,V,T
18991230,000000,1.1748149394989014,1.1750800609588623,1.1748149394989014,1.17507004737854,16,16 18991230,000000,1.1750950813293457,1.1751049757003784,1.1750850677490234,1.1751049757003784,4,4 18991230,000000,1.1751099824905396,1.1752899885177612,1.1751099824905396,1.1752899885177612,16,16 18991230,000000,1.1752749681472778,1.1752749681472778,1.1752500534057617,1.1752500534057617,6,6 18991230,000000,1.1752550601959229,1.1752550601959229,1.1752550601959229,1.1752550601959229,1,1 18991230,000000,1.1752400398254395,1.1752500534057617,1.1752300262451172,1.1752500534057617,4,4 18991230,000000,1.1752400398254395,1.1752400398254395,1.175225019454956,1.175225019454956,3,3 NORMAL EURUSD.csv utf-8[dos] 0% ㏑:1/5635752 ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Date,Time,O,H,L,C,V,T 18991230,000000,1548.800048828125,1548.800048828125,1548.800048828125,1548.800048828125,1,1 18991230,000000,1556.7000732421875,1556.7000732421875,1556.7000732421875,1556.7000732421875,1,1 18991230,000000,1552.0,1552.0,1552.0,1552.0,1,1 18991230,000000,1534.7000732421875,1534.7000732421875,1534.7000732421875,1534.7000732421875,1,1 18991230,000000,1595.0,1595.0,1595.0,1595.0,1,1 18991230,000000,1485.0,1485.0,1485.0,1485.0,1,2 18991230,000000,1503.0,1503.0,1503.0,1503.0,1,1 NORMAL GCQ21-COMEX.csv utf-8[dos] 0% ㏑:1/191884 ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Date,Time,O,H,L,C,V,T 18991230,000000,0.0,0.0,2623.0,2623.0,1,1 18991230,000000,0.0,0.0,2717.0,2717.0,1,1 18991230,000000,0.0,0.0,2750.0,2750.0,1,1 18991230,000000,0.0,2757.75,2742.5,2742.5,1,1 18991230,000000,0.0,2830.0,2805.75,2830.0,1,5 18991230,000000,0.0,2845.0,2814.75,2845.0,1,3 18991230,000000,0.0,2845.0,2814.75,2845.0,1,2 NORMAL ESM21-CME.csv utf-8[dos] 0% ㏑:1/191884 ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Is there a subset of SCID file you're using in your testing and maybe can share probably first 2-3 MB. Date Time Of Last Edit: 2021-06-19 06:49:55
|
| [2021-06-19 11:46:40] |
| Kiwi - Posts: 375 |
|
Its old code and it was working early 2020 but I notice it doesn't work on current data so SC presumably changed how they stored times. I will have a look at it and figure it out but it might take a week. I assume it relates to the 2151 change so it might be quite simple. I'll look tomorrow & if its easy I'll update, otherwise it could take a while. DateTime [Link] - [Top] The DateTime member variable is a SCDateTimeMS variable. This is a 64-bit integer. It represents the number of microseconds since the SCDateTime epoch of December 30, 1899. Versions prior to 2151, use a double precision floating point type variable to represent Date-Time values. For a complete explanation of the Date component of this value, refer to Date Value. The date value is the integer portion of the double. The fractional portion is the Time value which is represented as a fraction of one day where 1/86400000 is 1 ms. 86400000 is the number of milliseconds in a day. This Date and Time value is and must be in the UTC time zone. In Sierra Chart, if the Intraday Data Storage Time Unit in Global Settings >>Data/Trade Service Settings is set to a value greater than 1 Second like 1 Minute and the first trade within a new minute begins at a time after the beginning of the minute, for example 9:28:22, the time value part of the DateTime member in the record will be set to this time. A new record will begin if there is any trading on or after 9:29:00 and not 9:29:22. The data is aligned on time boundaries which are multiples of the Intraday Data Storage Time Unit. When writing tick by tick data to an Intraday data file, where each tick is a single record in the file, and multiple ticks/trades have the same timestamp, then it is acceptable for the Date-Time of the record to be the same timestamp as prior records. What Sierra Chart itself does in this case, is increment the microsecond portion of the Date-Time to create unique timestamps for each trade within a millisecond. Sierra Chart provides the /ACS_Source/SCDateTime.h file with various functions for working with these Date-Time values. Date Time Of Last Edit: 2021-06-19 11:51:54
|
| [2021-06-19 12:48:50] |
| biconoid - Posts: 5 |
|
Thank you Kiwi. No rush and as your time permits. Yes it seems related to newer implemtation to microseconds and highly unlikey but possible gap in documentation. It's good to know it's a genuine problem and a step in the right direction. I'll also try to read more into the Sierra header files, worth exploring. Date Time Of Last Edit: 2021-06-19 12:50:11
|
| [2021-06-19 14:47:59] |
| User907968 - Posts: 823 |
|
You wrote the information necessary to answer your question in the original message, but then used conflicting information when constructing your code. Trying to read Sierra Chart SCID file using Python and export as CSV/TSV Using following approach: With reference to s_IntradayRecord Structure Member Descriptions at Intraday Data File Format - The Intraday data record structure, s_IntradayRecord, is 40 bytes in size. - The Intraday data file header, s_IntradayHeader, is 56 bytes in size. - The DateTime member variable is a SCDateTimeMS variable. This is a 64-bit integer. It represents the number of microseconds since the SCDateTime epoch of December 30, 1899. "SCDateTimeMS variable. This is a 64-bit integer" which means you need to use 'q' as opposed to 'd' to unpack ('q4f4I' not 'd4f4I') - q being correct for 64-bit integer (long long) If you want the data in excel format, one method is to simply divide the datetime variable returned by the number of microseconds per day. For example, using the first value from your data.scid file, 3795467579000000 -> 43929.022905 -> 08/04/2020 00:32:59 |
| [2021-06-19 20:15:37] |
| biconoid - Posts: 5 |
|
Hi User907968, Thank you very much for your helpful and precise pointers. As a result i've tried to approach the problem below. Now quite close the output. - Date Part is working fine for all values provided from SCID. - Time part still is off by a little. Could you please check where could be the issue? """
------------------------------The DateTime member variable is a SCDateTimeMS variable. This is a 64-bit integer. It represents the number of microseconds since the SCDateTime epoch of December 30, 1899.""" import numpy as np def calculate_time_from(scid_date_time, zone=0): # Constant values sec_per_min, min_per_hour, hours_per_day = (60, 60, 24) sec_per_hour = sec_per_min * min_per_hour # 3600 sec_per_day = sec_per_hour * hours_per_day # 86400 u_sec_per_day = sec_per_day * 1000_000 # 86400_000_000 excel_epoch_time = np.datetime64('1899-12-30') # excel epoch date scid_date_time = int(scid_date_time) # d[0] value from struct.unpack('<q 4f 4I') time_zone = int(zone * min_per_hour) # UTC value, e.g +5, # Components to get full time time_elapsed = scid_date_time / u_sec_per_day # total time elapsed since Dec 30, 1899 date_part, time_part = str(time_elapsed).split('.') # split integer(date) and fraction(time) parts # Calculate delta in numpy_values delta_days = np.timedelta64(date_part, 'D') # timedelta in days for date_part delta_us = np.timedelta64(time_part, 'us') # timedelta in micro seconds for time_part delta_timezone = np.timedelta64(time_zone, 'm') # timedelta in minutes for time_zone my_time = excel_epoch_time + (delta_days + delta_us + delta_timezone) # final excel datetime value date_val, time_val = str(my_time).split('T') # individual date and time values return date_val, time_val # print(calculate_time_from(3795467579000000.0, -0.5)) # Output: ('2020-04-08', '00:08:10.509259') # print(calculate_time_from(3795467579000000.0, 0)) # Output: ('2020-04-08', '00:38:10.509259') # print(calculate_time_from(3795467579000000.0, 0.5)) # Output: ('2020-04-08', '01:08:10.509259') # print(f'Time: {my_time}') # Time: 2020-04-08T00:38:10.509259 # print(f'scid_date: {scid_date_time}') # scid_date: 3795467579000000 # print(f'time_elapsed: {time_elapsed}') # time_elapsed: 43929.02290509259 # print(f'date_part: {date_part}') # date_part: 43929 # print(f'time_part: {time_part}') # time_part: 02290509259 # print(f'np_units: {np.timedelta64(date_part)}') # np_units: 43929 generic time units # print(f'np_units: {np.timedelta64(time_part)}') # np_units: 2290509259 generic time units |
| [2021-06-19 23:20:51] |
| Kiwi - Posts: 375 |
|
Here is an update with a new dtformat (2) for data stored after SC2151. It could be modified to handle both but I don't use it so I've kept the old code and added a new branch when dtformat is set to 2 (as it is here). Let me know if it works with your files. I didn't worry about the sub second timestamps so they will all be the same timestamp (but with the data in the correct sequence) in the output csv file. Note that it won't work on older files unless you change the format in the covertRecords calls from 2 back to 1. """
Takes a SierraChart scid file (input argument 1) and converts it to a comma separated file (comma) Timezone conversion can follow the users local timezone, or a specified integer (input l or an integer but if the default filename is being used, '' must be specified for the filename) Inputs: filename & tz adjust (else UTC). if the filename is 'all' then all scids in directory are converted """ import csv import re import sys import os import struct from datetime import date, datetime, time, timedelta def deserialize(excelDateAndTime): """ Returns excel date time as Python datetime object """ date_tok = int(excelDateAndTime) time_tok = round(86400*(excelDateAndTime - date_tok)) d = date(1899, 12, 30) + timedelta(date_tok) t = time(*helper(time_tok, (24, 60, 60, 1000000))) return datetime.combine(d, t) def helper(factor, units): factor /= 86399.99 result = list() for unit in units: value, factor = divmod(factor * unit, 1) result.append(int(value)) result[3] = int(0) return result def datetimeTwo(excelDateAndTime, tzAdjust): """ Returns excel date time in format YYYYMMDD HHMMSS """ t = str(deserialize(excelDateAndTime + tzAdjust.seconds/86400.0)) return (re.sub(r'(:|-)', '', t)[:8], re.sub(r'(:|-)', '', t)[9:15]) def excelTimeAdjust(date_and_time, tzAdjust): return date_and_time + tzAdjust.seconds/86400.0 def convertRecords(filename, fileoutput, tzvar, dtformat): """ Read in records from SierraChart .scid data filename dtformat determines the datetime representation in column 1 (and 2) """ sizeHeader = 0x38 sizeRecord = 0x28 if tzvar == 99999: tzAdjust = datetime.fromtimestamp(0) - datetime.utcfromtimestamp(0) else: tzAdjust = timedelta(hours=tzvar) header = ('Date', 'Time', 'O', 'H', 'L', 'C', 'V', 'T') ftsv = open(fileoutput, 'w') fileout = csv.writer(ftsv, delimiter=',') with open(filename, 'rb') as fscid: fscid.read(sizeHeader) # discard header fileout.writerow(header) data = fscid.read(sizeRecord) while data: if dtformat == 2: # post SC2151 Microsecond timing dataRow = struct.unpack('q4f4I', data) date, time = datetimeTwo(dataRow[0]/86400000000, tzAdjust) outrow = (date, time, str(dataRow[1]), str(dataRow[2]), str(dataRow[3]), str(dataRow[4]), str(dataRow[5]), str(dataRow[6])) fileout.writerow(outrow) else: dataRow = struct.unpack('d4f4I', data) if dtformat == 0: adjustedTime = dataRow[0] + tzAdjust.seconds/86400.0 outrow = (str(adjustedTime), str(adjustedTime), str(dataRow[1]), str(dataRow[2]), str(dataRow[3]), str(dataRow[4]), str(dataRow[5]), str(dataRow[6])) fileout.writerow(outrow) elif dtformat == 1: date, time = datetimeTwo(dataRow[0], tzAdjust) outrow = (date, time, str(dataRow[1]), str(dataRow[2]), str(dataRow[3]), str(dataRow[4]), str(dataRow[5]), str(dataRow[6])) fileout.writerow(outrow) data = fscid.read(sizeRecord) return if __name__ == '__main__': filename = '/home/john/zRamdisk/Data/EUR.USD-CASH-IDEALPRO.scid' # filename = '/home/john/zRamdisk/Data/HSI-202106-HKFE.scid' tzvar = 0 if len(sys.argv) > 1: filename = sys.argv[1] if len(sys.argv) > 2: print(sys.argv[2], type(sys.argv[2])) if sys.argv[2] == 'l': tzvar = 99999 print(tzvar, type(tzvar)) else: tzvar = float(sys.argv[2]) print(tzvar, type(tzvar)) # filename = 'all' ############## remove ########################## if filename != "all": convertRecords(filename, filename[:-4] + 'csv', tzvar, 2) else: files = os.listdir(path='.') for filename in files: if filename[-5:] == '.scid': print("Converting this file", filename, "using Tz", tzvar) convertRecords(filename, filename[:-4] + 'csv', tzvar, 2) |
| [2021-09-03 09:41:21] |
| User155017 - Posts: 41 |
|
Hi, is there a way to generate automatic trades from spreadsheets using the custom fields added to the files for buy and sell signals? In general, how do you submit the order using the signal saved in the files generated in python? Thanks |
| [2021-09-25 18:27:00] |
| User155017 - Posts: 41 |
|
How do you send orders using python? You have written a c++ code to read the signal from file and submit from sierra? Do you have any example? |
| [2021-09-25 21:33:45] |
| Kiwi - Posts: 375 |
|
My order sending code has always been in C++ dlls to gain access to Sierra's rich interface. Communication of intent from Python can be by way of shared text files to pipe the commands in and responses out. Your dll can use these as a pipeline to SC's ordering api. |
| [2021-11-23 19:11:53] |
| User155017 - Posts: 41 |
|
Thanks. That is what I suspected. Is there a sample c++ code to look at which reads commands from files to send and modify orders? Could it be made general enough to manage several order independently? It would be nice to have a full api support to trade from a jupyter notebook, such as ibinsync for IB or web api of tradestation. |
| [2021-11-25 00:45:04] |
| Kiwi - Posts: 375 |
|
My code is completely focussed on my personal approach to this so, sorry, but I won't risk sharing it. However the ACSIL instructions are fairly clear and as long as you test what you're doing should work pretty well.
|
| [2022-01-15 09:26:26] |
| User155017 - Posts: 41 |
|
Using txt files to communicate with an external program is not as fast using the DTC server I suppose, but in absence of a working open source DTC client which would allow to trade directly from Python, it looks like the best practical solution. Using dtc would be much faster? I have some code in Python using multi-threading, could I use multi-threading in the dll to use in SC in case i would re-write it in c++? For now the file interface sounds the simplest solution, since I am not doing hft .. but I would like to know what could be the time gain in using a DTC server. Ideally if the dll can support multi-processing/threading it would be faster and no external communication would be required, but re-writing and debugging the code again is too much work for now. |
| [2022-01-15 23:11:43] |
| Kiwi - Posts: 375 |
|
I suspect the delay between SC calling the dll would be much greater than the time required to access a short file but file access time depends on what its stored on - mine are in my data directory which is on a linux ramdisk so that might not be much longer than the dtc access but if it was on a hdd that was spun down it could be quite slow. Just doing a python read of a short file off my ramdisk shows about 10 microseconds to open, read, and close, however a different method shows 100 microseconds, which is still fast. https://i.imgur.com/PPLKQ3s.png Edit: a later test in C suggests around 45 microseconds so the first reading must be an error due to my testing method. Date Time Of Last Edit: 2022-01-17 08:30:09
|
| [2022-01-16 14:32:33] |
| user2837 - Posts: 76 |
|
Thanks Kiwi, lots of great info here.
|
To post a message in this thread, you need to log in with your Sierra Chart account: