SmartQuant Discussion

Automated Quantitative Strategy Development, SmartQuant Product Discussion and Technical Support Forums
It is currently Mon Sep 23, 2019 2:07 pm

All times are UTC + 3 hours




Post new topic Reply to topic  [ 228 posts ]  Go to page Previous  1 ... 11, 12, 13, 14, 15, 16  Next
Author Message
 Post subject: Re: New IQFeed plug-in
PostPosted: Tue May 29, 2012 1:26 pm 
Offline

Joined: Mon Oct 04, 2010 5:13 pm
Posts: 353
Thanks a lot for clarification. Since we didn't encounter memory leak problems in historical data retrieval and we don't use real-time IQFeed I think we'll keep for now the previous version. We'll move to the next one with the new release of SQ (3.5.2 ?) when we know that the latest plugin is included there.

So for now we run 3.5.1 with the previous plugin in test environment.

Thanks
Ed


Top
 Profile  
 
 Post subject: Re: New IQFeed plug-in
PostPosted: Tue May 29, 2012 2:21 pm 
Offline

Joined: Wed May 05, 2010 9:49 pm
Posts: 497
edwardgonen wrote:
Thanks a lot for clarification. Since we didn't encounter memory leak problems in historical data retrieval and we don't use real-time IQFeed I think we'll keep for now the previous version. We'll move to the next one with the new release of SQ (3.5.2 ?) when we know that the latest plugin is included there.

So for now we run 3.5.1 with the previous plugin in test environment.

Thanks
Ed

In this case you must update only if you use alternative symbols.


Top
 Profile  
 
 Post subject: Re: New IQFeed plug-in
PostPosted: Tue May 29, 2012 2:26 pm 
Offline

Joined: Mon Oct 04, 2010 5:13 pm
Posts: 353
Thanks a lot for your great work!


Top
 Profile  
 
 Post subject: Re: New IQFeed plug-in
PostPosted: Mon Oct 08, 2012 10:51 am 
Offline

Joined: Sat Jan 22, 2011 1:40 am
Posts: 73
Two questions.

#1 - is there a way to turn off automatic Historical bar updates diring Live trading?

It seems OpenQuant updates Historical database with bars it generates from trades.

I.e. suppose I have a strategy with some symbols. Suppose I have 1-minute bars with "build bars from trades" enabled.
Then some of these bars will eventually appear in 1-minute bars in Data Manager.

Is there any way to turn off these database updates?

#2 - why updating historical database of 1-minute bars with existing data takes 100 times slower?

Suppose I load 1-minute bar data from 2012-01-01 until today through IQFeed. I get all this data at a
rate of less than 10-sec per symbol. 2 weeks later, if I want to add just 2 weeks - it will go a lot slower.
However, if I clear all the data and redownload from scratch - it would be faster.

It seems to be some kind of an issue with OpenQuant data storage architecture related to the .data file.
So if some data is present, OpenQuant checks/compares/seeks/rewrites or does something that it should
probably have an IF not to do and just replace the data.

Thanks


Top
 Profile  
 
 Post subject: Re: New IQFeed plug-in
PostPosted: Wed Oct 10, 2012 7:08 am 
Offline

Joined: Thu Jun 08, 2006 3:56 pm
Posts: 537
Location: BC Canada
I don't know the answer to #1, but the answer to #2 (as far as I know) is that the database is designed for fast reads, and normal speed adds, but slow replaces in the middle. This is because reading historical data is by far the most important use of the database.

If you have data to add/replace in the middle of a series, it can be faster to export the existing series to a CSV file (or multiple CSV files, actually) split on the dates where you want to add new data. Then reimport the first chunk, then your add-in chunk of data, then the remainder of the data after your chunk. That way you're always adding to the end of the db, and never asking it to insert bars into the middle of the series somewhere.

Probably the best way to do that is to practice on a small series, until you get the export/import scripts right. (At least, that's the way I would try to do it. For the record, I've never had to do what you seem to need to do, which is to add data into the middle of an existing series.) I hope this helps.


Top
 Profile  
 
 Post subject: Re: New IQFeed plug-in
PostPosted: Mon Nov 12, 2012 8:56 am 
Offline

Joined: Sat Jan 22, 2011 1:40 am
Posts: 73
kkkwj wrote:
I don't know the answer to #1, but the answer to #2 (as far as I know) is that the database is designed for fast reads, and normal speed adds, but slow replaces in the middle. This is because reading historical data is by far the most important use of the database.

If you have data to add/replace in the middle of a series, it can be faster to export the existing series to a CSV file (or multiple CSV files, actually) split on the dates where you want to add new data. Then reimport the first chunk, then your add-in chunk of data, then the remainder of the data after your chunk. That way you're always adding to the end of the db, and never asking it to insert bars into the middle of the series somewhere.

Probably the best way to do that is to practice on a small series, until you get the export/import scripts right. (At least, that's the way I would try to do it. For the record, I've never had to do what you seem to need to do, which is to add data into the middle of an existing series.) I hope this helps.


The problem is not adding in the middle. The problem is, let's say having data from 2012-09-01 to 2012-10-01 and then downloading data from 2012-10-02 to 2012-11-01. This will go SUPER SLOW. However, redownloading everything from 2012-09-01 to 2012-11-01 will go normal.


Top
 Profile  
 
PostPosted: Tue Nov 13, 2012 9:26 am 
Offline

Joined: Mon Oct 04, 2010 5:13 pm
Posts: 353
Hi,

Sorry that I'm reposting it here.

We moved from IQFeed client 4.8.1.7 to 4.9.0.3 (latest one) only on one of our computers - the one that retrieves 1500 US stocks daily historical data (1 bar = 1 day). It got stuck on TCO stock, after finishing about 80-90% of retrieval.

I'm attaching here the plug-in logs and recommend not to move to the new client until developer confirms that the new client is supported.

BTW - we run SQ 3.5.3 - the latest one.

Here is also the latest lines from SQ log:
Quote:
11/9/2012 11:49:44 PM DateTime: 11/9/2012 11:49:44 PM
Level: Critical
Source: Not defined.
Description: Cannot access a disposed object.
Object name: 'IDE'.
Details: System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IDE'.
at System.Windows.Forms.Control.MarshaledInvoke(Control caller, Delegate method, Object[] args, Boolean synchronous)
at System.Windows.Forms.Control.Invoke(Delegate method, Object[] args)
at System.Windows.Forms.Control.Invoke(Delegate method)
at OpenQuant.Runner.RunScenario()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
11/9/2012 11:49:46 PM Provider: IQFeed
Id: -1
Code: -1
Message: 23:49:46.952 [Log] *error*: Cannot access a closed file.

11/12/2012 12:46:10 AM ########################
11/12/2012 12:46:10 AM OS : Microsoft Windows NT 6.0.6002 Service Pack 2 32bit
11/12/2012 12:46:10 AM CPU(s): Intel(R) Core(TM)2 Duo CPU E6550 @ 2.33GHz
11/12/2012 12:46:10 AM RAM : 2052MB (1024+1024+4)
11/12/2012 12:46:10 AM CLR : 4.0.30319.269
11/12/2012 12:46:10 AM APP : OpenQuant 3.5.3 32bit
11/12/2012 12:46:13 AM Initializing DataManager...
11/12/2012 2:56:20 AM Provider: IQFeed
Id: -1
Code: -1
Message: [OnReceive] *error* <History>: An existing connection was forcibly closed by the remote host

11/12/2012 6:38:26 AM ########################
11/12/2012 6:38:26 AM OS : Microsoft Windows NT 6.0.6002 Service Pack 2 32bit
11/12/2012 6:38:26 AM CPU(s): Intel(R) Core(TM)2 Duo CPU E6550 @ 2.33GHz
11/12/2012 6:38:26 AM RAM : 2052MB (1024+1024+4)
11/12/2012 6:38:26 AM CLR : 4.0.30319.269
11/12/2012 6:38:26 AM APP : OpenQuant 3.5.3 32bit
11/12/2012 6:38:26 AM Initializing DataManager...
11/12/2012 7:26:48 AM Provider: IQFeed
Id: -1
Code: -1
Message: [OnReceive] *error* <History>: An existing connection was forcibly closed by the remote host

11/12/2012 10:24:07 PM Provider: IQFeed
Id: -1
Code: -1
Message: [OnReceive] *error* <History>: Non-negative number required.
Parameter name: byteCount




Thanks
Ed


Attachments:
IQFeed - Copy.zip [582.94 KiB]
Downloaded 314 times
Top
 Profile  
 
 Post subject: Re: New IQFeed plug-in
PostPosted: Sat Nov 17, 2012 12:18 am 
Offline

Joined: Mon Oct 04, 2010 5:13 pm
Posts: 353
Hi,

Any news on this issue?

Ed


Top
 Profile  
 
 Post subject: Re: New IQFeed plug-in
PostPosted: Sat Nov 17, 2012 9:13 am 
Offline
Site Admin

Joined: Thu Jul 17, 2003 10:39 am
Posts: 1478
Hi,

We will update IQFeed plugin next week.

_________________
SmartQuant Development Team


Top
 Profile  
 
 Post subject: Re: New IQFeed plug-in
PostPosted: Sat Nov 17, 2012 10:03 am 
Offline

Joined: Mon Oct 04, 2010 5:13 pm
Posts: 353
Hi,

Thanks a lot. Will the update be a part of new SQ Release or you are going to post it as a separate file?

Today I got notification about new SQ update but it seems I'll wait for the next week with upgrades as we have to run each upgrade for 1 week in simulation before production.

Aleksey - could you do us all a great favor and attach to each release announcement full (not short) list of changes and fixes? Every time you release a new version it is a dilemma to us whether replace our production. On one hand we want to run the latest SQ but then we don't want to make the changes to production environment. So if the list is full and detailed we can look through it and decide if we install it or skip.

Thanks
Ed


Top
 Profile  
 
 Post subject: Re: New IQFeed plug-in
PostPosted: Sat Nov 17, 2012 4:03 pm 
Offline
Site Admin

Joined: Thu Jul 17, 2003 10:39 am
Posts: 1478
Hi Ed,

I think that we will release updated plugin's assembly only. So, anyway, you will need to install the latest version of OQ.

And, we always recommend to use the latest version.

_________________
SmartQuant Development Team


Top
 Profile  
 
 Post subject: Re: New IQFeed plug-in
PostPosted: Sat Nov 17, 2012 5:30 pm 
Offline

Joined: Mon Oct 04, 2010 5:13 pm
Posts: 353
Aleksy,

If you release plug-in only there is a chance next release will not include the latest plug-in :). It happens to the best of us. But i understand your reasoning for that.

Ed


Top
 Profile  
 
 Post subject: Re: New IQFeed plug-in
PostPosted: Wed Nov 21, 2012 12:53 pm 
Offline

Joined: Sat Jan 22, 2011 1:40 am
Posts: 73
Can someone tell me why I see the following issue when I start my strategy?

Provider Errors:
[OnReceive] *error* <History>: Non-negative number required. Parameter name: byteCount


This worked fine yesterday with OpenQuant 3.5.1 64-bit. Stopped today. Downloaded new OpenQuant 3.5.4 64-bit. Still same issue. Using IQFeed 4.9.0.3 for data for the past few days.

Requests for historical data are:

1. To get last few days of 1-minute bars:

DateTime datetime2 = Clock.Now;
DateTime datetime1 = datetime2.AddDays(-10);
BarSeries my_bars = GetHistoricalBars("IQFeed", datetime1, datetime2, 60);

2. To get last few daily bars:

DateTime datetime2 = Clock.Now.AddDays(-1);
datetime2 = new DateTime(datetime2.Year, datetime2.Month, datetime2.Day);
DateTime datetime1 = datetime2.AddDays(-60);
BarSeries my_bars = GetHistoricalBars("IQFeed", datetime1, datetime2, 24 * 60 * 60);

This worked fine for several months before today...

Thanks.


Top
 Profile  
 
 Post subject: Re: New IQFeed plug-in
PostPosted: Wed Nov 21, 2012 1:10 pm 
Offline

Joined: Mon Oct 04, 2010 5:13 pm
Posts: 353
Hi

I suggest you downgrade to the previous iqfeed client 4.8 or something


Top
 Profile  
 
 Post subject: Re: New IQFeed plug-in
PostPosted: Wed Nov 21, 2012 5:43 pm 
Offline

Joined: Sat Jan 22, 2011 1:40 am
Posts: 73
Thanks for your suggestion. However, I just uninstalled 4.9.0.3 and installed 4.8.1.7 back and still get exact same error.

Any other suggestions? Is there a newer IQFeed plugin that may fix it? What is a "byteCount" parameter to IQFeed and why is it negative?


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 228 posts ]  Go to page Previous  1 ... 11, 12, 13, 14, 15, 16  Next

All times are UTC + 3 hours


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group