streamReader for server URLs
Asked Answered
R

0

3

I've been working with the code form this answer, provided by Martin R. The code is awesome and it is very useful. However, it doesn't work with the links, while working fine with files. After putting some NSLogs and breaks, I have actually found that problem is in this code block:

init?(path: String, delimiter: String = "\n", encoding: UInt = NSUTF8StringEncoding, chunkSize : Int = 4096) {
    self.chunkSize = chunkSize
    self.encoding = encoding

    self.fileHandle = NSFileHandle(forReadingFromURL: NSURL(string: path)!, error: nil)
    println("PATH IS \(path)")
    println("FILE HANDLE IS \(fileHandle)")
    if self.fileHandle == nil {
        println("FILE HANDLE IS NIL!")
        return nil
    }

The code above actually contains some minor changes, compared with Martin's answer. Also Apple says that it is possible to use fileHandle with forReadingFromURL and it shouldn't return nil. But here is the console output:

PATH IS http://smth.com
   FILE HANDLE IS nil
   FILE HANDLE IS NIL!!!!!

The question is what's wrong?

UPDATE

As Martin R has kindly explained to me, this code won't work with URLs, this answer states the same, so I have rewritten the code, guiding by previous answers:

import Foundation
import Cocoa

class StreamReader  {

let encoding : UInt
let chunkSize : Int
var atEof : Bool = false
var streamData : NSData!
var fileLength : Int
var urlRequest : NSMutableURLRequest
var currentOffset : Int
var streamResponse : NSString

var fileHandle : NSFileHandle!
let buffer : NSMutableData!
let delimData : NSData!

var reponseError: NSError?
var response: NSURLResponse?

init?(path: NSURL, delimiter: String = "\n", encoding: UInt = NSUTF8StringEncoding, chunkSize : Int = 10001000) {
    println("YOUR PATH IS \(path)")

    self.chunkSize = chunkSize
    self.encoding = encoding
    self.currentOffset = 0
    urlRequest = NSMutableURLRequest(URL: path)
    streamData = NSURLConnection.sendSynchronousRequest(urlRequest, returningResponse:&response, error:&reponseError)
    streamResponse = NSString(data:streamData!, encoding:NSUTF8StringEncoding)!
    self.fileLength = streamData.length
    //println("WHAT IS STREAMDATA \(streamData)")
    //println("WHAT IS URLREQUEST \(urlRequest)")

    if streamData == nil {
        println("LINK HAS NO CONTENT!!!!!")
    }

            self.fileLength = streamResponse.length
    println("FILE LENGTH IS \(fileLength)")
    self.buffer = NSMutableData(capacity: chunkSize)!

    // Create NSData object containing the line delimiter:
    delimData = delimiter.dataUsingEncoding(NSUTF8StringEncoding)!
    println("WHAT DOES THE DELIMITER \(delimiter)LOOK LIKE?")
    println("WHAT IS DELIMDATA \(delimData)")
}

deinit {
    self.close()
}

/// Return next line, or nil on EOF.
func nextLine() -> String? {

    if atEof {
        println("AT THE END OF YOUR FILE!!!")
        return nil
    }
    // Read data chunks from file until a line delimiter is found:

    if currentOffset >= fileLength {
        return nil
    }
    var blockLength : Int = buffer.length

    var range = buffer.rangeOfData(delimData, options: NSDataSearchOptions(0), range: NSMakeRange(currentOffset, blockLength))
            //println("STREAM DATA \(streamData)")

    println("RANGE IS \(range)")
    while range.location == NSNotFound {
        var nRange = NSMakeRange(currentOffset, chunkSize)
        println("nRange is \(nRange)")
        var tmpData = streamData.subdataWithRange(nRange)
        //println("TMP data length \(tmpData.length)")
        currentOffset += blockLength
        //println("TMPDATA is \(tmpData)")
        if tmpData.length == 0 {
            // EOF or read error.
            println("ERROR ????")
            atEof = true
            if buffer.length > 0 {
                // Buffer contains last line in file (not terminated by delimiter).
                let line = NSString(data: buffer, encoding: encoding);
                buffer.length = 0
                println("THE LINE IS \(line)")
                return line
            }
            // No more lines.
            return nil
        }
        buffer.appendData(tmpData)
        range = buffer.rangeOfData(delimData, options: NSDataSearchOptions(0), range: NSMakeRange(0, buffer.length))
    }

    // Convert complete line (excluding the delimiter) to a string:
    let line = NSString(data: buffer.subdataWithRange(NSMakeRange(0, range.location)),
        encoding: encoding)
    // Remove line (and the delimiter) from the buffer:
    buffer.replaceBytesInRange(NSMakeRange(0, range.location + range.length), withBytes: nil, length: 0)

    return line
}

/// Start reading from the beginning of file.
func rewind() -> Void {
    //streamData.seekToFileOffset(0)
    buffer.length = 0
    atEof = false
}

/// Close the underlying file. No reading must be done after calling this method.
func close() -> Void {
    if streamData != nil {
        streamData = nil
    }
}
}

extension StreamReader : SequenceType {
func generate() -> GeneratorOf<String> {
    return GeneratorOf<String> {
        return self.nextLine()
    }
}
}

But actually this code is very far from being perfect and I would like to see any recommendations on improving it. Please, be kind. I am very amateur and very inexperienced( but sooner or later I will learn it)

Finally, it is working. And probably the last problem is left, the code doesn't stop, it continues to read file from the beginning.

So now the question is probably more close to 'What's wrong with my code?', compared with previous: 'What's wrong?'

UPDATE

I have rewritten last parts of code like this:

let line = NSString(data: buffer.subdataWithRange(NSMakeRange(0, range.location + 1)),
        encoding: encoding)
    buffer.replaceBytesInRange(NSMakeRange(0, range.location + range.length), withBytes: nil, length: 0)
    println("COMPLETE LINE IS \(line)")
    if line!.containsString("\n"){
        println("CONTAINS NEW LINE")
        //println("BUFFER IS \(buffer)")
        //
        println("COMPLETE LINE IS \(line)")
        return line
    }
    else {

        println("NO LINE!")
        atEof == true
        return nil
    }

The idea is to go through all the lines which contain \n and to exclude one line, which is the last one and shouldn't have the \n. But! Despite the fact that I have checked non-printing characters and there was no \n, here is the surprising console output: Optional("lastline_blablabla\n") Probably now the question is, how to stop at the last line even if it contains \n?

If you need to retrieve data from the URL my code above (the one under first update) will work

But my own problem with \n has several ways to be solved. One of which I used in my code. As it's very individual I won't post the solution to \n at the end of file issue. Also I am not sure that my both solutions for urlStreamReader and \n are the best, so if you advice any better solution it will be much appreciated by me and probably some other people.

Great thanks to Martin R, who explained a lot, wrote great code and was very nice

Rutter answered 31/10, 2014 at 11:57 Comment(9)
A missing slash in the URL?Bangor
@Martin R No, the URL is fine, I just have replaced the original one with this text. Also I have tried with different links, ftps, https.Rutter
NSFileHandle works only with files, sockets, pipes, and devices. It cannot read via http. You probably have to use NSURLConnection or NSURLSession.Bangor
@Martin What about apple's notes?? Do I understand it all wrong? Return Value The initialized file handle object or nil if no file exists at url. Discussion The file pointer is set to the beginning of the file. You cannot write data to the returned file handle object. Use the readDataToEndOfFile or readDataOfLength: methods to read data from it. When using this method to create a file handle object, the file handle owns its associated file descriptor and is responsible for closing it.Rutter
The first paragraph states: "You use file handle objects to access data associated with files, sockets, pipes, and devices.". I am quite sure that NSFileHandle(forReadingFromURL:) works only with file URLs. (Reading from a http URL is an asynchronous process, that's what NSURLConnection and NSURLSession are made for.)Bangor
Seems I have found the same: "It doesn't read from a servers URL, which led me to this approach (I would like it for people to edit this for me, as it needs lazy loading, optimization, etc.)" From this [question] (#3916242)Rutter
The motivation for StreamReader was that a (huge) file can be read line-by-line, without loading the entire file into memory. – Your (new) code reads the complete remote file into memory with streamData = NSURLConnection.sendSynchronousRequest(...). This may take a time and may block the current thread.Bangor
@Martin so there is no way to fix it? I mean, I am working with it for quite a long time and don't want to give up, because it loads text lines beautifully. I have fixed even the issue with the chunkSize and it's quite an achievement for me. I just need to make it see the end of text.Rutter
It can probably be fixed and I will have a look at it later. But note that streamResponse already contains the entire remote file as a string, so let lines = streamResponse.componentsSeparatedByString("\n") would be an array of all lines. So where is the advantage of dividing the data in chunks here? (But +1 for not giving up :)Bangor

© 2022 - 2024 — McMap. All rights reserved.