How can I extract data from DAT and IDX files of SCADA CIMPLICITY software?

Question

I am tasked with extracting the data from the data files of an old software - CIMplicity HMI Plant Edition version 6.0. Its a SCADA software from 2002. I have a copy of the data files directory which contains a lot of *.DAT and *.IDX files. I am required to extract this data to CSV or to an SQL database. Some of the DAT files are just plain-text, but others have a binary-like format and when opened in PSPad, shows up in HEX view mode.

What tools can I use to reliably read and extract the data from these files?

TIA.

UPDATE: I've added a directory listing of the directory with the data files:

 Directory of C:\tmp\xxxxxxII\data

04/30/2013  01:53 PM    <DIR>          .
04/30/2013  01:53 PM    <DIR>          ..
09/02/2008  10:46 AM            17,260 1220323606.clz
09/02/2008  10:46 AM            60,490 1220323607.clz
09/10/2008  06:36 PM           288,554 1220323608.clz
09/02/2008  10:46 AM            66,977 1220323609.clz
09/10/2008  06:37 PM            23,900 1220323610.clz
09/10/2008  06:37 PM            19,162 1220323611.clz
09/10/2008  06:48 PM            37,596 1220323612.clz
09/10/2008  06:49 PM            27,882 1220323613.clz
09/10/2008  06:49 PM            47,850 1220323614.clz
09/10/2008  06:50 PM            47,816 1220323615.clz
09/10/2008  06:52 PM         3,427,511 1220323616.clz
09/02/2008  10:46 AM            31,169 1220323617.clz
09/10/2008  06:53 PM            30,353 1220323618.clz
09/02/2008  10:46 AM           122,159 1220323619.clz
09/02/2008  10:50 AM         3,539,414 1220323828.clz
09/10/2008  06:02 PM               208 action.dat
09/10/2008  06:02 PM             3,072 action.idx
02/19/2002  11:58 PM             5,636 alarm_class.dat
02/19/2002  11:58 PM             3,072 alarm_class.idx
09/23/2008  04:26 PM           137,128 alarm_def.dat
09/23/2008  04:26 PM            49,152 alarm_def.idx
02/19/2002  11:58 PM             2,929 alarm_field.dat
02/19/2002  11:58 PM             4,096 alarm_field.idx
02/19/2002  11:58 PM                 0 alarm_intproc.dat
02/19/2002  11:58 PM             4,096 alarm_intproc.idx
02/19/2002  11:58 PM               135 alarm_mgr.dat
02/19/2002  11:58 PM             3,072 alarm_mgr.idx
09/23/2008  04:26 PM            69,531 alarm_routing.dat
09/23/2008  04:26 PM           387,072 alarm_routing.idx
02/19/2002  11:58 PM               912 alarm_type.dat
02/19/2002  11:58 PM             3,072 alarm_type.idx
02/19/2002  11:58 PM                 0 alm_setup.dat
02/19/2002  11:58 PM             4,096 alm_setup.idx
02/19/2002  11:58 PM                 0 alm_setup_cl.dat
02/19/2002  11:58 PM             3,072 alm_setup_cl.idx
02/19/2002  11:58 PM                 0 alm_setup_fr.dat
02/19/2002  11:58 PM             3,072 alm_setup_fr.idx
02/19/2002  11:58 PM                 0 alm_user.dat
02/19/2002  11:58 PM             3,072 alm_user.idx
02/19/2002  11:58 PM                 0 alrm_blk_alarm.dat
02/19/2002  11:58 PM             4,096 alrm_blk_alarm.idx
02/19/2002  11:58 PM                 0 alrm_blk_group.dat
02/19/2002  11:58 PM             3,072 alrm_blk_group.idx
02/11/1998  04:05 PM               602 amlp.cfg
09/10/2008  06:53 PM             2,415 class.dat
09/10/2008  06:53 PM             3,072 class.idx
02/11/1998  04:06 PM                 5 client.cfg
09/10/2008  02:14 PM               393 comm_exe.dat
09/10/2008  02:14 PM             4,096 comm_exe.idx
09/23/2008  03:40 PM             9,893 datalog.dat
09/23/2008  03:40 PM             5,120 datalog.idx
02/19/2002  11:58 PM             1,272 data_field.dat
02/19/2002  11:58 PM             3,072 data_field.idx
09/04/2008  03:10 PM             1,218 dbms_def.dat
09/04/2008  03:10 PM             3,072 dbms_def.idx
09/16/2008  10:45 AM            37,820 derived_point.dat
09/16/2008  10:45 AM            16,384 derived_point.idx
09/10/2008  02:14 PM               256 devcom_proc.dat
09/10/2008  02:14 PM             4,096 devcom_proc.idx
09/10/2008  02:16 PM             1,305 device.dat
09/10/2008  02:16 PM             5,120 device.idx
09/23/2008  04:26 PM         2,243,024 device_point.dat
09/23/2008  04:26 PM         1,745,920 device_point.idx
09/23/2008  04:04 PM                 6 dyn_cfg.cfg
02/19/2002  11:58 PM                 0 em_alu.dat
02/19/2002  11:58 PM             3,072 em_alu.idx
02/19/2002  11:58 PM                 0 es_eu_conv.dat
02/19/2002  11:58 PM             3,072 es_eu_conv.idx
02/19/2002  11:58 PM                 0 es_point_info.dat
02/19/2002  11:58 PM             4,096 es_point_info.idx
09/23/2008  04:26 PM           719,712 eu_conv.dat
09/23/2008  04:26 PM            78,848 eu_conv.idx
09/10/2008  06:02 PM               166 event.dat
09/10/2008  06:02 PM             3,072 event.idx
09/10/2008  06:03 PM               121 event_action.dat
09/10/2008  06:03 PM             3,072 event_action.idx
04/30/2013  01:53 PM                 0 f.txt
02/19/2002  09:49 PM           199,302 field_def.dat
02/19/2002  09:49 PM            87,040 field_def.idx
09/10/2008  02:15 PM             1,608 fr.dat
09/10/2008  02:15 PM             5,120 fr.idx
07/15/2010  03:41 PM               262 gef_cfg.ini
09/23/2008  03:39 PM             6,435 glb_parms.dat
09/23/2008  03:39 PM             6,144 glb_parms.idx
12/15/1999  11:16 AM               899 ie_deflds.cfg
11/14/2001  11:06 AM             1,101 ie_formats.cfg
02/19/2002  09:49 PM             7,548 keyconst.dat
02/19/2002  09:49 PM            18,432 keyconst.idx
02/19/2002  09:49 PM            16,984 key_field.dat
02/19/2002  09:49 PM            13,312 key_field.idx
02/19/2002  09:49 PM             9,546 lenconst.dat
02/19/2002  09:49 PM            17,408 lenconst.idx
09/10/2008  02:14 PM               990 logproc.dat
09/10/2008  02:14 PM             3,072 logproc.idx
09/23/2008  03:54 PM            47,952 log_attrib.dat
09/23/2008  03:54 PM            77,824 log_attrib.idx
09/23/2008  03:40 PM             1,848 log_event.dat
09/23/2008  03:40 PM             4,096 log_event.idx
08/05/1998  09:04 AM             1,671 log_names.cfg
09/10/2008  02:14 PM               121 master.mcp
07/18/2008  06:32 PM                32 master_mcp.app
09/10/2008  02:14 PM                29 master_mcp.dc
07/18/2008  06:32 PM                52 master_mcp.rp
09/28/2001  02:22 PM            17,449 master_opc_0.ini
02/19/2002  11:58 PM            11,312 meas_assoc.dat
02/19/2002  11:58 PM             8,192 meas_assoc.idx
02/19/2002  11:58 PM               276 meas_system.dat
02/19/2002  11:58 PM             3,072 meas_system.idx
02/19/2002  11:58 PM             1,096 meas_unit.dat
02/19/2002  11:58 PM             3,072 meas_unit.idx
09/10/2008  02:14 PM               365 model.dat
09/10/2008  02:14 PM             4,096 model.idx
07/18/2008  06:32 PM                86 node.dat
07/18/2008  06:32 PM             3,072 node.idx
09/10/2008  02:14 PM             2,167 node_logproc.dat
09/10/2008  02:14 PM             5,120 node_logproc.idx
09/23/2008  04:26 PM            32,890 object.dat
09/23/2008  04:26 PM            28,672 object.idx
09/23/2008  04:26 PM           310,464 object_attrib.dat
09/23/2008  04:26 PM           293,888 object_attrib.idx
09/23/2008  04:26 PM            22,080 object_routing.dat
09/23/2008  04:26 PM            30,720 object_routing.idx
09/10/2008  02:14 PM               715 physproc.dat
09/10/2008  02:14 PM             5,120 physproc.idx
04/26/2010  12:27 PM         2,527,608 point.dat
04/26/2010  12:27 PM           637,952 point.idx
02/19/2002  11:58 PM                95 point_alstr.dat
02/19/2002  11:58 PM             3,072 point_alstr.idx
02/19/2002  11:58 PM                 0 point_disp.dat
02/19/2002  11:58 PM             3,072 point_disp.idx
02/19/2002  11:58 PM               194 point_enum.dat
02/19/2002  11:58 PM             3,072 point_enum.idx
02/19/2002  11:58 PM               216 point_enum_fld.dat
02/19/2002  11:58 PM             4,096 point_enum_fld.idx
02/19/2002  11:58 PM               609 point_type.dat
02/19/2002  11:58 PM             3,072 point_type.idx
09/10/2008  02:14 PM               129 port_comm.dat
09/10/2008  02:14 PM             4,096 port_comm.idx
09/10/2008  02:14 PM                42 port_list.dat
09/10/2008  02:14 PM             4,096 port_list.idx
09/10/2008  02:14 PM               294 port_type.dat
09/10/2008  02:14 PM             3,072 port_type.idx
02/11/1998  04:05 PM                 5 projects.cfg
09/10/2008  02:14 PM               123 protocol.dat
09/10/2008  02:14 PM             3,072 protocol.idx
02/19/2002  11:58 PM                37 ptmgmt.dat
02/19/2002  11:58 PM             3,072 ptmgmt.idx
04/27/2010  03:25 PM            19,343 ptx_points.cfg
02/19/2002  11:58 PM                 0 pt_uf_assoc.dat
02/19/2002  11:58 PM             4,096 pt_uf_assoc.idx
02/19/2002  11:58 PM                93 pt_uf_sets.dat
02/19/2002  11:58 PM             3,072 pt_uf_sets.idx
02/19/2002  11:58 PM             1,568 pt_user_fields.dat
02/19/2002  11:58 PM             4,096 pt_user_fields.idx
02/19/2002  09:49 PM             9,856 record_def.dat
02/19/2002  09:49 PM            13,312 record_def.idx
02/19/2002  11:58 PM                 0 redund_addrs.dat
02/19/2002  11:58 PM             4,096 redund_addrs.idx
02/19/2002  11:58 PM               111 role.dat
02/19/2002  11:58 PM             3,072 role.idx
02/19/2002  11:58 PM               159 role_subsys.dat
02/19/2002  11:58 PM             4,096 role_subsys.idx
09/10/2008  02:14 PM             1,067 service.dat
09/10/2008  02:14 PM             5,120 service.idx
02/19/2002  11:58 PM               181 service_use.dat
02/19/2002  11:58 PM             3,072 service_use.idx
09/23/2008  03:57 PM               711 TCPIP0.ini
02/19/2002  11:58 PM                 0 trend_pt.dat
02/19/2002  11:58 PM             3,072 trend_pt.idx
09/02/2008  09:29 AM               434 user.dat
09/02/2008  09:29 AM             4,096 user.idx
09/10/2008  02:15 PM               784 user_fr.dat
09/10/2008  02:15 PM             6,144 user_fr.idx
07/15/2010  03:40 PM                15 warmdata.sav
             174 File(s)     17,990,662 bytes

UPDATE: Screenshot of point.dat attached:

Supplementary

Without seeing a sample of the file(s), I'm not sure what kind of help are you expecting? – Overcharge

How about a sample of the file itself? The directory listing doesn't help. Normally idx is for "index" of a .dat (data) file. That's about all I can help you with given what you have provided so far. – Overcharge

Do you want to write a tool, buy a tool, or find a tool? – Lordosis

@Beta: Ideally I need to extract the data with an available tool. If there isn't a tool available then I need to know how to extract the data using Delphi or any other means. – Supplementary

If it helps, BeyondCompare has a converter module - thefoolonthehill.net/drupal/Beyond%20Compare%20Rules – Dialecticism

Ok, sorry, just thought it worth a try ... – Dialecticism

Legacy data files written by software now extinct, in a binary format that may be proprietary. This is starting to look like an exercise in cryptanalysis. I'd hunt around for a more modern version of Cimplicity, parse its files with BeyondCompare, reverse-engineer the format, then scan the old files assuming that the older format is similar and perhaps a little simpler. – Lordosis

@Lordosis chances are slim that a tools exists, this is such a specialized field. – Blas

@Lordosis 2002 that is recent for SCADA, chances are you can still get it to run. Data from a tool used and deployed in the 80s that starts to be called legacy in SCADA circles. Then again a "modern" version of the tool may cost something around 10k, so that may be out of question. – Blas

@BurhanKhalid The current version of CIMPLICITY is suing MSSQL. It is not far fetched that it actually is some databse, the question is, which? – Blas

Answer 1

My condolences, you have been tasked with an almost impossible task.

Ok, I don't know much about CIMPLICITY HMI Plant Edition, save what I can see from the outside. But maybe I can give you some helpful pointers, since I am also developing industrial HMI and Engineering software. (WinCC & PCS7)

First off, the format is, with a certainty of 99%, a proprietary format. The only real help you can expect to get is from GE. The question is, will they give it to you and at what cost. (Since you are trying to extract data, you are probably migrating away; then why should they help.)

So some reverse engineering is probably in order. You want the software that created the project. That is the engineering package and not the HMI run-time software. (May be one piece of software.) Your company should still have licenses for that from the original project.

From there you either create a new project and see what happens and or load the existing project and then play around and see what changes in the data. It is important that you understand what data is in the system and where you can find it.

It is important to note that the data comes in two forms, project configuration and run-time data.

The project configuration is actually not data, this is "just" configuration what the run-time system should display/react. The project data are things like users, views/screens, PLC variable bindings or control scripts. I honestly don't know how you want to put most of this efficiently into CVS or a database. But it is possible WinCC uses MSSQL. (with a default password, thank you stuxnet)

The run-time data is simpler. This are either event logs, that record things like operator commands, alarms and warnings or recorded values. This data should be easily extracted, since it is strictly formatted. To find out what and where was recorded you should open the project in the engineering software, there you should find clues about logging and trends.

Nevertheless you want to get yourself a good hex editor and LOTS of time.

Addendum:

After you added the listing and screenshot, the files make 3/4 sense to me.

For example, alarm_class.* contains the clases of alarms that the run-time may raise, the alarm_type the data types of alarms, alarm_field contains the configured alarms for field alaram, i.e. form the PLC, alarm_rounting are routing or network errors, alm_user are probably user alarms, i.e. from scripts in the HMI runtime.

Everything with point in the name is probably a "measurement point", that is a field device; either a sensor or a feedback from an actuator.

Everything with user, is probably the configured users and their permissions.

redund_addrs is a map of PLC or device addresses that are redundant siblings of primary values.

Everything with port is probably about the "ports" on the SCADA server or PLC, for example FF, PROFIBUS or PROFINET.

Everything with object is probably structures. That is when single variables (aka tags) are bound together for form hierarchical values. For example the value of a sensor and all values about it's status are bound into one structure, this is maybe then treated as one measurement point.

Also when looking at the hex dump you provided, it seams like a structured constant length format.

In this case:

struct Point
{
    char name[16] = "$ALARM.ACKED";
    char type[32] = "UDINT";
    char comment[128] = "...";
};

The values are then padded with space "0x20".

Addendum 2:

You may actually looking at a flat file database, such like dBase and each *.dat is the data and *.idx the table index. Just an idea. It may pay to invest some time into DB technology available circa 2000. Then maybe you can just "dump" the data.

Blas

Thanks for the long reply. Its not Dbase format, I tried opening the files with a couple of DBF viewers and they errorred out. Many of the files are plain-text format, with the text being delimited according to column-position. – Supplementary

Maybe it is codebase. That has *.dat and *.idx files; just saw it in one project. But no guarantees there. – Blas

Answer 2

::EXPORT:: If you open up a command prompt in the master directory Type "idtpop event" for event.idx file It will create a file event.idt
This file can be edited in notepad or other editor.

Any .DAT & .IDX files of Cimplicity can be accessed in the same way.

::IMPORT: If you want the required changes to be imported, at the command prompt type scpop event
This will regenerate the event file. Also do the other files.

Acidulate

Answer 3

Nova Gladson is correct, but that only works on a system that has Cimpicity installed on it.

Cimplicity is still being actively developed and marketed. The newest version is very capable of opening version 6.

Cimplicity HMI & Scada

Brobdingnagian