PHP - Import CSV file to mysql database Using LOAD DATA INFILE
Asked Answered
S

2

16

I have a .csv file data like that

Date,Name,Call Type,Number,Duration,Address,PostalCode,City,State,Country,Latitude,Longitude
"Sep-18-2013 01:53:45 PM","Unknown","outgoing call",'123456',"0 Secs","null","null","null","null","null",0.0,0.0,,,
"Sep-18-2013 01:54:14 PM","Unknown","outgoing call",'1234567890',"0 Secs","null","null","null","null","null",0.0,0.0,,,
"Sep-18-2013 01:54:37 PM","Unknown","outgoing call",'14772580369',"1 Secs","null","null","null","null","null",0.0,0.0,,,

and I'm using the following code to insert the data into database

$sql = "LOAD DATA INFILE `detection.csv`
              INTO TABLE `calldetections`
              FIELDS TERMINATED BY '".@mysql_escape_string(",").
             "` OPTIONALLY ENCLOSED BY `".@mysql_escape_string("\"").
             "` OPTIONALLY ENCLOSED BY `".@mysql_escape_string("\'").
             "` ESCAPED BY `".@mysql_escape_string("\\").
              "` LINES TERMINATED BY `".",,,\\r\\n".
             "`IGNORE 1 LINES `"

             ."(`date`,`name`,`type`,`number`,`duration`,`addr`,`pin`,`city`,`state`,`country`,`lat`,`log`)";
      $res = @mysql_query($con,$sql); 

but nothing is inserted; where is the mistake?

Scissors answered 20/9, 2013 at 10:58 Comment(0)
H
47

If you'd do echo($sql); before you execute it you'd see that syntax of your query is incorrect for following reasons:

  1. Filename should be enclosed in quotes rather than backticks because it's a string literal not an identifier.

  2. There is absolutely no need to call mysql_escape_string() to specify a delimiter in FIELDS TERMINATED BY and ENCLOSED BY and ESCAPED BY clauses.

  3. You overuse backticks. In fact in your case, since there are no reserved words used, you ditch them all. They only add clutter.

  4. At the end of the very first line of your CSV file you have to have ,,, because you use them as part of a line delimiter. If you won't do that you'll skip not only first line but also second one that contains data.

  5. You can't use ENCLOSED BY clause more than once. You have to deal with Number field in a different way.

  6. Looking at your sample rows IMHO you don't need ESCAPED BY. But if you feel like you need it use it like this ESCAPED BY '\\'.

That being said a syntacticly correct statement might look like this

LOAD DATA INFILE 'detection.csv'
INTO TABLE calldetections
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"' 
LINES TERMINATED BY ',,,\r\n'
IGNORE 1 LINES 
(date, name, type, number, duration, addr, pin, city, state, country, lat, log)

Now IMHO you need to transform quite a few fields while you load them:

  1. if date in your table is of datetime data type then it needs to be transformed, otherwise you'll get an error

    Incorrect datetime value: 'Sep-18-2013 01:53:45 PM' for column 'date' at row

  2. you have to deal with single qoutes around values in Number field

  3. you most likely want to change "null" string literal to actual NULL for addr, pin, city, state, country columns

  4. if duration is always in seconds then you can extract an integer value of seconds and store it that way in your table to be able to easily aggregate duration values later.

That being said a useful version of the statement should look something like this

LOAD DATA INFILE 'detection.csv'
INTO TABLE calldetections
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"' 
LINES TERMINATED BY ',,,\r\n'
IGNORE 1 LINES 
(@date, name, type, @number, @duration, @addr, @pin, @city, @state, @country, lat, log)
SET date = STR_TO_DATE(@date, '%b-%d-%Y %h:%i:%s %p'),
    number = TRIM(BOTH '\'' FROM @number),
    duration = 1 * TRIM(TRAILING 'Secs' FROM @duration),
    addr = NULLIF(@addr, 'null'),
    pin  = NULLIF(@pin, 'null'),
    city = NULLIF(@city, 'null'),
    state = NULLIF(@state, 'null'),
    country = NULLIF(@country, 'null') 

Below is the result of executing the query on my machine

mysql> LOAD DATA INFILE '/tmp/detection.csv'
    -> INTO TABLE calldetections
    -> FIELDS TERMINATED BY ','
    -> OPTIONALLY ENCLOSED BY '"' 
    -> LINES TERMINATED BY ',,,\n'
    -> IGNORE 1 LINES 
    -> (@date, name, type, @number, @duration, @addr, @pin, @city, @state, @country, lat, log)
    -> SET date = STR_TO_DATE(@date, '%b-%d-%Y %h:%i:%s %p'),
    ->     number = TRIM(BOTH '\'' FROM @number),
    ->     duration = 1 * TRIM(TRAILING 'Secs' FROM @duration),
    ->     addr = NULLIF(@addr, 'null'),
    ->     pin  = NULLIF(@pin, 'null'),
    ->     city = NULLIF(@city, 'null'),
    ->     state = NULLIF(@state, 'null'),
    ->     country = NULLIF(@country, 'null');
Query OK, 3 rows affected (0.00 sec)
Records: 3  Deleted: 0  Skipped: 0  Warnings: 0

mysql> select * from calldetections;
+---------------------+---------+---------------+-------------+----------+------+------+------+-------+---------+------+------+
| date                | name    | type          | number      | duration | addr | pin  | city | state | country | lat  | log  |
+---------------------+---------+---------------+-------------+----------+------+------+------+-------+---------+------+------+
| 2013-09-18 13:53:45 | Unknown | outgoing call | 123456      |        0 | NULL | NULL | NULL | NULL  | NULL    | 0.0  | 0.0  |
| 2013-09-18 13:54:14 | Unknown | outgoing call | 1234567890  |        0 | NULL | NULL | NULL | NULL  | NULL    | 0.0  | 0.0  |
| 2013-09-18 13:54:37 | Unknown | outgoing call | 14772580369 |        1 | NULL | NULL | NULL | NULL  | NULL    | 0.0  | 0.0  |
+---------------------+---------+---------------+-------------+----------+------+------+------+-------+---------+------+------+
3 rows in set (0.00 sec)

And finally in php assigning a query string to $sql variable should look like this

$sql = "LOAD DATA INFILE 'detection.csv'
        INTO TABLE calldetections
        FIELDS TERMINATED BY ','
        OPTIONALLY ENCLOSED BY '\"' 
        LINES TERMINATED BY ',,,\\r\\n'
        IGNORE 1 LINES 
        (@date, name, type, @number, @duration, @addr, @pin, @city, @state, @country, lat, log)
        SET date = STR_TO_DATE(@date, '%b-%d-%Y %h:%i:%s %p'),
            number = TRIM(BOTH '\'' FROM @number),
            duration = 1 * TRIM(TRAILING 'Secs' FROM @duration),
            addr = NULLIF(@addr, 'null'),
            pin  = NULLIF(@pin, 'null'),
            city = NULLIF(@city, 'null'),
            state = NULLIF(@state, 'null'),
            country = NULLIF(@country, 'null') ";
Hysterectomy answered 22/9, 2013 at 7:29 Comment(8)
Thank you so mach for your detailed answer.Scissors
@Scissors You're quite welcome. Good luck :) Don't hesitate to ask a follow up question if needed.Hysterectomy
@Hysterectomy What a great, clear answer. If you aren't a very well paid teacher you should seriously consider it. Thanks.Deepseated
<?php include '../class/pdo.php'; $pdo = new PDO(); $pdo_connection = $pdo->pdo_connect(); $tt="1cs2cs1cs17409061651452795086two.csv"; $aa=","; $bb='"'; $cc='\n'; echo $query = "LOAD DATA LOCAL INFILE '".$tt."' INTO TABLE data2 FIELDS TERMINATED BY '".$aa."' OPTIONALLY ENCLOSED BY '".$bb."' LINES TERMINATED BY '".$cc."' IGNORE 1 LINES;"; $queryresource = $pdo_connection->query($query); ?> I am trying to use same way as suggested it only inserted 2 rows instead the csv file have 1 million rows.Victorvictoria
@abhinavBruteForce Did you try to run it outside php and look for MySQL errors and warning? Most likely than not you have problems with your data (incorrect/incompatible type, missing value/value that violates not null/foreign constraints, etc.)Hysterectomy
yes peterm i have echo the query and executed directly in mysql console then also it uploaded data but only two rows with out any error. Please suggest where i am doing wrongVictorvictoria
Query OK, 1 row affected, 65535 warnings (16.36 sec) Records: 1048576 Deleted: 0 Skipped: 1048575 Warnings: 13631488 why records are getting skipped. Here also only two records getting inserted.Victorvictoria
@abhinavBruteForce There you you hate it 65535 warnings. Run SHOW WARNINGS right after your run LOAD DATA and see what the problems you have with your data.Hysterectomy
M
1

Insert bulk more than 7000000 record in 1 minutes in database(superfast query with calculation)

    mysqli_query($cons, '
    LOAD DATA LOCAL INFILE "'.$file.'"
    INTO TABLE tablename
    FIELDS TERMINATED by \',\'
    LINES TERMINATED BY \'\n\'
    IGNORE 1 LINES
    (isbn10,isbn13,price,discount,free_stock,report,report_date)
     SET RRP = IF(discount = 0.00,price-price * 45/100,IF(discount = 0.01,price,IF(discount != 0.00,price-price * discount/100,@RRP))),
         RRP_nl = RRP * 1.44 + 8,
         RRP_bl = RRP * 1.44 + 8,
         ID = NULL
    ')or die(mysqli_error());
    $affected = (int) (mysqli_affected_rows($cons))-1; 
    $log->lwrite('Inventory.CSV to database:'. $affected.' record inserted successfully.');

RRP and RRP_nl and RRP_bl is not in csv but we are calculated that and after insert that.

Malocclusion answered 17/2, 2017 at 7:56 Comment(1)
Welcome to Stack Overflow! Please don't add the same answer to multiple questions. Answer the best one and flag the rest as duplicates. See Is it acceptable to add a duplicate answer to several questions?Effie

© 2022 - 2024 — McMap. All rights reserved.