How do I query raw data from a Proficy Historian?

Question

How can I retrieve raw time-series data from a Proficy Historian/iHistorian?

Ideally, I would ask for data for a particular tag between two dates.

Electroscope

Answer 1

There are several different sampling modes you can experiment with.

• Raw
• Interpolated
• Lab
• Trend
• Calculated

These modes are available using all of the following APIs.

• User API (ihuapi.dll)
• SDK (ihsdk.dll)
• OLEDB (iholedb.dll)
• Client Acess API (Proficy.Historian.ClientAccess.API)

Of these the trend sampling mode is probably what you want since it is specifically designed for charting/trending. Though, lab and interpolated may be useful as well.

Read the electronic book for more information on each sampling mode. On my machine it is stored as C:\Program Files\GE Fanuc\Proficy Historian\Docs\iHistorian.chm and I have version 3.5 installed. Pay particular attention to the following sections.

• Using the Historian OLE DB Provider
• Advanced Topics | Retrieval

Here is how you can construct an OLEDB to do trend sampling.

set 
    SamplingMode = 'Trend',
    StartTime = '2010-07-01 00:00:00',
    EndTime = '2010-07-02 00:00:00',
    IntervalMilliseconds = 1h
select 
    timestamp, 
    value, 
    quality 
from 
    ihRawData 
where 
    tagname = 'YOUR_TAG'

Showing the equivalent methods using the User API and the SDK are complex (more so with the User API) since they require a lot of plumbing in the code to get setup. The Client Access API is newer and uses WCF behind the scenes.

By the way, there are a few limitations with the OLEDB method though.

• Despite what the documentation says I have never been able to get native query parameters to work. That is a showstopper if you want to use it with SQL Server Reporting Services for example.
• You cannot write samples into the archive or in any way make changes to the Historian configuration including adding/changing tags, writing messages, etc.
• It can be a little slow in some cases.
• It has no provision for crosstabbing multiple tagnames into the columns and then carrying forward samples so that a value exists for each timestamp and tag combination. The trend sampling mode gets you halfway there, but still does not crosstab and does not actually load raw samples. Then again the User API and SDK cannot do this either.

Tish

A little late I know, but better late than never. Plus, answering old questions on SO is kind of my style anyway. – Tish

I don't have the time to test this right now but it's exactly what I was looking for when I started. Thanks! Can you by chance add a note about where the ebook is typically located? – Electroscope

Any ideas about how to use the built in historian functions like PreviousValue in this way? Can the oledb do this? something along the lines of select "timestamp, PreviousValue(Time, tagname) from ..." – Lugsail

@roviuser: I don't think so, but post another question and clearly describe what you want and I might be able to help. – Tish

Have a look at my answer Brian – Coldhearted

Stumbled upon this thread after banging my head against the wall trying to figure out why OleDBParameters don't seem to work with the ihOLEDB provider. Apparently this is still broken 13 years later. – Anybody

Answer 2

A coworker of mine put this together:

In web.config:

<add name="HistorianConnectionString" 
     providerName="ihOLEDB.iHistorian.1" 
     connectionString="
       Provider=ihOLEDB.iHistorian;
       User Id=;
       Password=;
       Data Source=localhost;"
/>

In the data layer:

public DataTable GetProficyData(string tagName, DateTime startDate, DateTime endDate)
{
    using (System.Data.OleDb.OleDbConnection cn = new System.Data.OleDb.OleDbConnection())
    {
        cn.ConnectionString = webConfig.ConnectionStrings.ConnectionStrings["HistorianConnectionString"];
        cn.Open();

        string queryString = string.Format(
                "set samplingmode = rawbytime\n select value as theValue,Timestamp from ihrawdata where tagname = '{0}' AND timestamp between '{1}' and '{2}' and value > 0 order by timestamp",
                tagName.Replace("'", "\""), startDate, endDate);

        System.Data.OleDb.OleDbDataAdapter adp = new System.Data.OleDb.OleDbDataAdapter(queryString, cn);
        DataSet ds = new DataSet();

        adp.Fill(ds);
        return ds.Tables[0];
    }
}

---

Update:

This worked well but we ran into an issue with tags that don't update very often. If the tag didn't update near the start or end of the requested startDate and endDate, the trends would look bad. Worse, still were cases where there were no explicit points during the window requested--we'd get no data back.

I resolved this by making three queries:

1. The previous value before the start-date
2. The points between startDate and endDate
3. The next value after the endDate

This is a potentially inefficient way to do it but It Works:

public DataTable GetProficyData(string tagName, DateTime startDate, DateTime endDate)
{
    DataSet ds = new DataSet();
    string queryString;
    System.Data.OleDb.OleDbDataAdapter adp;

    using (System.Data.OleDb.OleDbConnection cn = new System.Data.OleDb.OleDbConnection())
    {
        cn.ConnectionString = proficyConn.ConnectionString;
        cn.Open();

        // always get a start value
        queryString = string.Format(
             "set samplingmode = lab\nselect value as theValue,Timestamp from ihrawdata where tagname = '{0}' AND timestamp between '{1}' and '{2}' order by timestamp",
            tagName.Replace("'", "\""), startDate.AddMinutes(-1), startDate);
        adp = new System.Data.OleDb.OleDbDataAdapter(queryString, cn);
        adp.Fill(ds);

        // get the range
        queryString = string.Format(
             "set samplingmode = rawbytime\nselect value as theValue,Timestamp from ihrawdata where tagname = '{0}' AND timestamp between '{1}' and '{2}' order by timestamp",
            tagName.Replace("'", "\""), startDate, endDate);
        adp = new System.Data.OleDb.OleDbDataAdapter(queryString, cn);
        adp.Fill(ds);

        // always get an end value
        queryString = string.Format(
             "set samplingmode = lab\nselect value as theValue,Timestamp from ihrawdata where tagname = '{0}' AND timestamp between '{1}' and '{2}' order by timestamp",
        tagName.Replace("'", "\""), endDate.AddMinutes(-1), endDate);
        adp = new System.Data.OleDb.OleDbDataAdapter(queryString, cn);
        adp.Fill(ds);

        return ds.Tables[0];
    }
}

And yes, I know, those queries should be parameterized.

Electroscope

OLE is very slow. Using the API has much better performance, you can also write extension methods that allow you to do much more than OLE will support out of the box. – Coldhearted

Regarding getting no samples back you can do "ReadSamplesByCount" instead of time, that way you get at least 1 sample back. If there are not many samples the sample however may be years after the time period you want to display. – Coldhearted

Answer 3

We wrote a wrapper DLL that looked like this like this:

[DllImport("IHUAPI.dll", CallingConvention = CallingConvention.StdCall, EntryPoint = "ihuReadRawDataByTime@24")]
public static extern int ihuReadRawDataByTime(int serverhandle, string tagname, ref IHU_TIMESTAMP startTime, ref IHU_TIMESTAMP endTime, ref int noOfSamples, ref IHU_DATA_SAMPLE* dataValues);
...
private int _handle;

public HistorianTypes.ErrorCode ReadRawByTime(string tagName, DateTime startTime, DateTime endTime,
                                              out double[] timeStamps, out double[] values, out IhuComment [] comments)
{
    var startTimeStruct = new IhuApi.IHU_TIMESTAMP();  //Custom datetime to epoch extension method
    var endTimeStruct = new IhuApi.IHU_TIMESTAMP();

    int lRet = 0;
    int noOfSamples = 0;
    startTimeStruct = DateTimeToTimeStruct(dstZone.ToUniversalTime(startTime));
    endTimeStruct = DateTimeToTimeStruct(dstZone.ToUniversalTime(endTime));
    IhuApi.IHU_DATA_SAMPLE* dataSample = (IhuApi.IHU_DATA_SAMPLE*)new IntPtr(0);

    try {
        lRet = IhuApi.ihuReadRawDataByTime
            (
                _handle, // the handle returned from the connect
                tagName, // the single tagname to retrieve
                ref startTimeStruct, // start time for query
                ref endTimeStruct, // end time for query
                ref noOfSamples, // will be set by API
                ref dataSample // will be allocated and populated in the user API
            );
            ....

Some notes are that iFIX will check if the DLL is loaded on startup so you need to do things like dynamically load/unload the DLL so that other applications don't crash. We did this by deleting/adding registry keys on the fly.

Another one is if you poll 10,000 samples and 1 of the samples are corrupted it will drop all 10,000 samples. You need to implement a bad data handler that will start at either side of the bad data and increment in steps to get all data either side of the bad sample.

There are several C header files that contain all of the error codes and the function header for the DLL.

Coldhearted

Answer 4

Michael--in IP21 there is an "Interpolated" table, as well as the "actual" data point table. Does Proficy have that as well?

Parol

@Parol I believe they do, though, in my case I needed raw data – Electroscope

I ran into the same thing a few years ago with a PI system, so I do feel your pain! – Parol