How to use async await in python child process

Question

i am getting this error 2021-03-16T06:52:01.205519+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=POST path="/formsubmit" host=test-searcher.herokuapp.com request_id=fc6ee612-693f-4e87-9aa6-331989fd05fd fwd="117.197.157.92" dyno=web.1 connect=0ms service=30001ms status=503 bytes=0 protocol=https in my node js application. To resolve this issue i want to use async/await but i m not able to get to use that in my js code. please help me in this.

as far i think this is happening becoz of heroku's policy of 30 sec https://devcenter.heroku.com/articles/limits#http-timeouts

My code of node js is below

// form submit request
app.post('/formsubmit', function(req, res){

    csvData = req.files.csvfile.data.toString('utf8');
    filteredArray = cleanArray(csvData.split(/\r?\n/))
    csvData = get_array_string(filteredArray)
    // Send request to python script
    var spawn = require('child_process').spawn;
    var process = spawn('python', ["./webextraction.py", csvData, req.body.keywords, req.body.full_search])

    process.stdout.on('data', async function(data){

      var data_array = []
      data_array.push(data.toString())
      res.setHeader('Content-disposition', 'attachment; filename=output.csv');
      res.set('Content-Type', 'text/csv');
      var strin = data_array[0].trim()
      strin = strin.replace(/(\r)/gm, "");
      res.send(strin)
   

    });

});

I m getting the below error on heroku

2021-03-16T06:51:07.059126+00:00 heroku[web.1]: Restarting
2021-03-16T06:51:07.087873+00:00 heroku[web.1]: State changed from up to starting
2021-03-16T06:51:08.797707+00:00 heroku[web.1]: Stopping all processes with SIGTERM
2021-03-16T06:51:08.989681+00:00 heroku[web.1]: Process exited with status 143
2021-03-16T06:51:28.040420+00:00 heroku[web.1]: Starting process with command `node app.js`
2021-03-16T06:51:30.263324+00:00 app[web.1]: server running on port 3000
2021-03-16T06:51:30.478212+00:00 heroku[web.1]: State changed from starting to up
2021-03-16T06:51:38.000000+00:00 app[api]: Build succeeded
2021-03-16T06:52:01.205519+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=POST path="/formsubmit" host=test-searcher.herokuapp.com request_id=fc6ee612-693f-4e87-9aa6-331989fd05fd fwd="117.197.157.92" dyno=web.1 connect=0ms service=30001ms status=503 bytes=0 protocol=https
2021-03-16T06:53:07.169061+00:00 heroku[router]: at=info method=GET path="/" host=test-searcher.herokuapp.com request_id=db24efd1-cd75-4cb2-a00e-b72cd7160f4e fwd="117.197.157.92" dyno=web.1 connect=0ms service=17ms status=200 bytes=3208 protocol=https

I tried below to solve this issue

// form submit request
app.post('/formsubmit', function(req, res){

    csvData = req.files.csvfile.data.toString('utf8');
    filteredArray = cleanArray(csvData.split(/\r?\n/))
    csvData = get_array_string(filteredArray)
    // Send request to python script
    var spawn = require('child_process').spawn;
    var process = spawn('python', ["./webextraction.py", csvData, req.body.keywords, req.body.full_search])

    process.stdout.on('data', async function(data){

      var data_array = []
      
      await data_array.push(data.toString())

      res.setHeader('Content-disposition', 'attachment; filename=output.csv');
      res.set('Content-Type', 'text/csv');
      var strin = data_array[0].trim()
      strin = strin.replace(/(\r)/gm, "");
      res.send(strin)
});
});

but i am still getting the same error

Restarting
2021-03-16T07:50:07.577176+00:00 heroku[web.1]: State changed from up to starting
2021-03-16T07:50:08.520994+00:00 heroku[web.1]: Stopping all processes with SIGTERM
2021-03-16T07:50:08.642371+00:00 heroku[web.1]: Process exited with status 143
2021-03-16T07:50:26.333143+00:00 heroku[web.1]: Starting process with command `node app.js`
2021-03-16T07:50:28.628842+00:00 app[web.1]: server running on port 3000
2021-03-16T07:50:29.106381+00:00 heroku[web.1]: State changed from starting to up
2021-03-16T07:50:44.000000+00:00 app[api]: Build succeeded
2021-03-16T07:53:02.758211+00:00 heroku[router]: at=info method=GET path="/" host=test-searcher.herokuapp.com request_id=29b18518-800e-4ec6-a3f5-7d579e056394 fwd="117.197.136.20" dyno=web.1 connect=1ms service=23ms status=200 bytes=3208 protocol=https
2021-03-16T07:53:55.638541+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=POST path="/formsubmit" host=test-searcher.herokuapp.com request_id=18a16fd9-1670-426e-a998-940ec3a16a1b fwd="117.197.136.20" dyno=web.1 connect=0ms service=30008ms status=503 bytes=0 protocol=https

Eleni

I'm curious about what you're trying to achieve with the python child process. if you're trying to parse a CSV file there are plenty of node packages to do the job (ex:csv2json) – Elusion

in this data there is csv string, this child process calls python script which is taking more than 30sec to return output and this heroku terminates process becoz it thinks it as synchronous call – Eleni

when using an async function you must be awaiting for some promise to be resolved. in your code, I think you should add await data_array.push(data.toString()) – Elusion

@AbdelKaderGlt the thing is that i want to wait for response time from my python script which i think, using Async/ await can be achieved – Eleni

@AbdelKaderGlt, i tried but same error again – Eleni

@AbdelKaderGlt, when to use then in code after await – Eleni

@AbdelKaderGlt, please see into this and guide me – Eleni

@AbdelKaderGlt, thers is issue on heroku platform devcenter.heroku.com/articles/limits#http-timeouts – Eleni

@AbdelKaderGlt r u there – Eleni

@eol still i m not able to solve this issue – Eleni