SmartQuant Discussion

Automated Quantitative Strategy Development, SmartQuant Product Discussion and Technical Support Forums
It is currently Fri Dec 13, 2019 2:42 pm

All times are UTC + 3 hours




Post new topic Reply to topic  [ 10 posts ] 
Author Message
PostPosted: Thu Oct 17, 2019 10:39 am 
Offline

Joined: Mon Sep 02, 2019 10:31 am
Posts: 27
Hi Team,

My requirement is to get historical bars for multiple instruments simultaneously (in parallel) using IQFeed. For example, if I subscribe for two instruments I1 and I2 then bars sequence should be I1.Bar1, I2.Bar1, I1.Bar2, I2.Bar2, ... I1.BarN, I2.BarN.

When I call this method for multiple instruments by calling this method multiple times in sequence, it returns bars (trigger Bar event) in the same sequence as I called the method. It would return all bars for I1 then all bars for I2 and so on.

Code:
framework.DataManager.GetHistoricalBars(
  comboBoxProvider.SelectedItem.ToString(),
  framework.InstrumentManager[symbol],
  dateTimePickerFrom.Value, dateTimePickerTo.Value,
  BarType.Time, Convert.ToInt64(textBoxBarSize.Text)
);


I also tried an alternate way using BarFactory as below:

Code:
Framework framework = Framework.Current;
IHistoricalDataProvider historicalDataProvider = framework.ProviderManager.GetHistoricalDataProvider("IQFeed");
historicalDataProvider.Connect();
framework.DataSimulator.SubscribeBar = false;

framework.DataSimulator.DateTime1 = dateTimePickerFrom.Value;
framework.DataSimulator.DateTime2 = dateTimePickerTo.Value;

var symbols = new string[] { "AAPL", "GOOGL" };

foreach (var symbol in symbols)
{
  framework.EventManager.BarFactory.Add(framework.InstrumentManager[symbol], BarType.Time, Convert.ToInt64(textBoxBarSize.Text), providerId: historicalDataProvider.Id);
}


Could you please suggest what I might be missing?

Thanks,


Top
 Profile  
 
PostPosted: Thu Oct 17, 2019 1:11 pm 
Offline

Joined: Wed May 05, 2010 9:49 pm
Posts: 562
Parallel requests mean async responses, this is obvious.
From the other hand, you cannot do parallel request from single thread, so they will be queued.
Therefore, even if you get all bar series, you must sync series manually.


Top
 Profile  
 
PostPosted: Fri Oct 18, 2019 4:27 pm 
Offline

Joined: Mon Sep 02, 2019 10:31 am
Posts: 27
Can the framework not do this multi-threading for us and provide the data in same way as it is provided to Startegy? When I run (backtest) BollingerBands with AAPL and MSFT, logs from OnBar event are as below:

Code:
Bar AAPL: 204.51
Bar MSFT: 136.115
Bar AAPL: 204.58
Bar MSFT: 136.55
Bar AAPL: 204.81
Bar MSFT: 136.365
Bar AAPL: 204.75
Bar AAPL: 204.82
Bar MSFT: 136.44
Bar AAPL: 205.17
Bar AAPL: 205.16
Bar MSFT: 136.72
Bar AAPL: 205.25
Bar MSFT: 136.68


As you can see, the data appears in parallel, one bar from AAPL and then one from MSFT and so on. This is exactly what we require but we do not want to create Startegy. We want to do it directly using Framework's object. I was hoping if there is way to receive data in the form like above independently using framework's object. I think BarFactory should be able to achieve this but I am not sure what flags or settings I should set.


Top
 Profile  
 
PostPosted: Fri Oct 18, 2019 5:13 pm 
Offline

Joined: Wed May 05, 2010 9:49 pm
Posts: 562
Our framework processes all queued events in single thread. So this is not parallel.


Top
 Profile  
 
PostPosted: Fri Oct 18, 2019 8:35 pm 
Offline

Joined: Mon Sep 02, 2019 10:31 am
Posts: 27
Can you suggest any workaround? Do we have to create a separate framework's instance in a separate thread for each instrument if we want to receive data in parallel as I mentioned in my post earlier?


Top
 Profile  
 
PostPosted: Mon Oct 21, 2019 10:23 am 
Offline

Joined: Wed May 05, 2010 9:49 pm
Posts: 562
I suppose parallel requests will be look like:

Code:
class Program
    {
        static string[] symbols = { "AAPL", "IBM", "MSFT", "CSCO", "AMD" };
        static Dictionary<string, BarSeries> bars = new Dictionary<string, BarSeries>();
        static void Main(string[] args)
        {
            Framework framework = new Framework("main");

            IHistoricalDataProvider historicalDataProvider = framework.ProviderManager.GetHistoricalDataProvider("IQFeed");
            historicalDataProvider.Connect(5000); //init connection for creating iqfeed gateway

            Task[] tasks = new Task[symbols.Length];

            for (int i = 0; i < symbols.Length; i++)
            {
                string symbol = symbols[i];

                tasks[i] = new Task(() => GetBars(symbol));
                tasks[i].Start();
            }

            Task.WaitAll(tasks);

            Console.WriteLine("All tasks completed");

            Console.ReadLine();
        }

        static void GetBars(string symbol)
        {
            bars.Add(symbol, null);

            Framework framework = new Framework("framework_" + symbol);

            IHistoricalDataProvider historicalDataProvider = framework.ProviderManager.GetHistoricalDataProvider("IQFeed");
            historicalDataProvider.Connect(5000);
           
            Instrument instrument = framework.InstrumentManager[symbol];

            if (instrument == null)
                return;

            Console.WriteLine($"Sending request for {symbol}");

            BarSeries series = framework.DataManager.GetHistoricalBars(historicalDataProvider, instrument, DateTime.Now.AddDays(-100), DateTime.Now, BarType.Time, 60);

            if (series == null)
                Console.WriteLine("Series is null for {symbol}");
            else
                Console.WriteLine($"Got series with {series.Count} elements for {symbol}");

            bars[symbol] = series;
        }
    }


You can process all bars after
Code:
Task.WaitAll(tasks);


Top
 Profile  
 
PostPosted: Wed Oct 23, 2019 8:35 pm 
Offline

Joined: Mon Sep 02, 2019 10:31 am
Posts: 27
What if I need to read data from the framework's local storage not IQFeed? Can framework return data in the format (simultaneously not sequence) that I mentioned in previous post, in this case, using some built-in methods?


Top
 Profile  
 
PostPosted: Fri Oct 25, 2019 9:54 am 
Offline

Joined: Wed May 05, 2010 9:49 pm
Posts: 562
harsh wrote:
What if I need to read data from the framework's local storage not IQFeed? Can framework return data in the format (simultaneously not sequence) that I mentioned in previous post, in this case, using some built-in methods?

No, as I said earlier, you have to process and synchronize all the data yourself.


Top
 Profile  
 
PostPosted: Sat Nov 02, 2019 2:02 pm 
Offline

Joined: Mon Sep 02, 2019 10:31 am
Posts: 27
We need to read historical market data for about 600 instruments at a time in parallel (not in sequence).

It really doesn't seem a practical solution to create 600 instances of the Framework to achieve this. I am sceptical that doing so will hog CPU/Memory of the machine.

If possible, can you share APIs for reading directly from market data storage (binary/database) so we can read directly from the storage?

Or, can you suggest any other solution for this scenario? This requirement is critical to our systems.

Thanks,


Top
 Profile  
 
PostPosted: Mon Nov 11, 2019 9:37 am 
Offline

Joined: Wed May 05, 2010 9:49 pm
Posts: 562
harsh wrote:
We need to read historical market data for about 600 instruments at a time in parallel (not in sequence).

It really doesn't seem a practical solution to create 600 instances of the Framework to achieve this. I am sceptical that doing so will hog CPU/Memory of the machine.

If possible, can you share APIs for reading directly from market data storage (binary/database) so we can read directly from the storage?

Or, can you suggest any other solution for this scenario? This requirement is critical to our systems.

Thanks,

I made several tests, but unfortunately I can not advise optimal way to load data in parallel for this moment.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 10 posts ] 

All times are UTC + 3 hours


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group