Send and receive objects through sockets in Python
Asked Answered
D

3

6

I have searched a lot on the Internet, but I haven't been able to find the solution to send an object over the socket and receive it as is. I know it needs pickling which I have already done. And that converts it to bytes and is received on the other hand. But how can I convert those bytes to that type of object?

process_time_data = (current_process_start_time, current_process_end_time)
prepared_process_data = self.prepare_data_to_send(process_time_data)
data_string = io.StringIO(prepared_process_data)
data_string =  pack('>I', len(data_string)) + data_string
self.send_to_server(data_string)

This is the code which is converting the object to StringIO on the client and sending to the server. And on the server side I am getting bytes. Now I am searching for bytes to be converted to StringIO again so that I can get the object value.

In the code, Object is wrapped in StringIO and is being sent over the socket. Is there a better approach?

The server-side code is as follows.

server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
#server.setblocking(0)
server.bind(('127.0.0.1', 50000))
server.listen(5)
inputs = [server]
outputs = []
message_queues = {}

while inputs:
    readable, writeable, exceptional = select.select(inputs, outputs, inputs)
    for s in readable:
        if s is server:
            connection, client_address = s.accept()
            print(client_address)
            connection.setblocking(0)
            inputs.append(connection)
            message_queues[connection] = queue.Queue()
            print('server started...')
        else:
            print('Getting data step 1')
            raw_msglen = s.recv(4)
            msglen = unpack('>I', raw_msglen)[0]
            final_data = b''
            while len(final_data) < msglen:
                data = s.recv(msglen - len(final_data))
                if data:
                    #print(data)
                    final_data += data
                    message_queues[s].put(data)
                    if s not in outputs:
                        outputs.append(s)
                    else:
                        if s in outputs:
                            outputs.remove(s)
                else:
                    break
            inputs.remove(connection)
            #s.close()
            del message_queues[s]

            process_data = ProcessData()
            process_screen = ProcessScreen()

            if final_data is not None:
                try:
                    deserialized_data = final_data.decode("utf-8")
                    print(deserialized_data)
                except (EOFError):
                    break
            else:
                print('final data is empty.')

            print(process_data.project_id)
            print(process_data.start_time)
            print(process_data.end_time)
            print(process_data.process_id)

The two helper functions are as follows:

def receive_all(server, message_length, message_queues, inputs, outputs):
    # Helper function to recv message_length bytes or return None if EOF is hit
    data = b''
    while len(data) < message_length:
        packet = server.recv(message_length - len(data))
        if not packet:
            return None
        data += packet
        message_queues[server].put(data)
        if server not in outputs:
            outputs.append(server)
        else:
            if server in outputs:
                outputs.remove(server)
    inputs.remove(server)
    del message_queues[server]
    return data


def receive_message(server, message_queues, inputs, outputs):
    # Read message length and unpack it into an integer
    raw_msglen = receive_all(server, 4, message_queues, inputs, outputs)
    if not raw_msglen:
        return None
    message_length = unpack('>I', raw_msglen)[0]

    return receive_all(server, message_length, message_queues, inputs, outputs)

And two of the model classes are as follows:

class ProcessData:
    process_id = 0
    project_id = 0
    task_id = 0
    start_time = 0
    end_time = 0
    user_id = 0
    weekend_id = 0

# Model class to send image data to the server
class ProcessScreen:
    process_id = 0
    image_data = bytearray()
Dup answered 20/11, 2017 at 12:13 Comment(12)
Please take some time to read the help page, especially the sections named "What topics can I ask about here?" and "What types of questions should I avoid asking?". And more importantly, please read the Stack Overflow question checklist. You might also want to learn about Minimal, Complete, and Verifiable Examples.Schlessel
@SudheeshSinganamalla Thanks for your all caring advices. But do you actually understand what I asked. And where am I stuck with ?Dup
@SudheeshSinganamalla once my teacher said no question is bad. It is always the answer. So you should ask what comes to your mind. Leave the answers to the understanding of the answering person.Dup
Can anybody let me know why are they voting this question down. If you are not able to understand the question. please move on. But that shouldn't let anybody vote down.Dup
@Dup "once my teacher said no question is bad" - they are wrong, there are definitely bad questions. This is one of them. See suggested duplicate below for what information is needed when asking a question, and also for the answer that will help you.Chic
As @SudheeshSinganamalla said, your example of the problem should be minimal, complete, and verifiable. This is not a complete example; it is not a standalone code block: that is, it cannot be run as is. Without the respective definitions, the only context we have for the variables and functions being used is their names. This makes it very difficult to answer the question at all, let alone adequately, which is why it's being downvoted: The purpose of the vote is a kind of triage to rank questions according to priority. This may or may not be a bad question, but it's certainly poorly asked.Nonaligned
@Bilkokuya Now what should I say on this? Don't want to be offensive. But you should read my question clearly. I have said I need to send and receive objects and not simple text. yeah but it clearly shows that you havent read my question clearly. Now please show me some code for that and then say your beautiful words. And my teacher was right. Question is still good but no body knows the answer just like me. And just trying to show the expertise. Show me code buddies the question still remains unanswered.Dup
@Dup You need to give us more to work with. You can edit the question to address the concerns I and other users commenting here have expressed: Chiefly, include the rest of the example code (complete), minus any superfluous blocks that are not relevant to the question (minimal) with a single repeatable issue that we can address (verifiable). Otherwise, you are not likely to get an answer.Nonaligned
@ErickShepherd i am not able to stop laughing that whatever I say or edit question but all are now boarded to the same train that vote down this question and constantly marking it down. I think more than enough data is provided even though the question was already clear. This should not happen this way. Even people are marking your comments up. Just even not trying to see the question. Guys if you really know something then tell but or atleast comment. But just marking down and down. This is ridiculousDup
@Bilkokuya exactly. I am stuck there only. And any help would be highly appreciated. And if you make fun of me I am happy for that. But voting down negative without understanding is not right. Just imagine SudheeshSinganamalla who started all this hasn't even commented back or put any light on the situation just like you are doing. So there is a difference between an inspector and a helper. I appreciate that you are trying to help me. And yes you are on the right point. I am stuck with that only.Dup
You seem to have made comments where you've asked me to stay away. If you had edited to question to look remotely as it looks now when I made my comment, you'd have received an answer in the meanwhile. The reason why we moderate is so that someone experienced can answer your question but when your question is not clear or takes up way too much time, no one would spend time to answer it. Good luck!Schlessel
Well @SudheeshSinganamalla would you be able to answer it now if it is clear? I believe the point is crystal clear what I am asking. Instead responding to my last comment you could have answered the question or could have said that it is out of my knowledge and I accept moderation but it should be given a bit of time to put some point in there. I believe you guys have improved the manner in which the question is being presented now. But still I am waiting for any clue. And that definitely doesn't mean that I am sitting idle and doing nothing. I am also putting my efforts.Dup
S
28

You're looking for pickle and the loads and dumps operations. Sockets are basically byte streams. Let us consider the case you have.

class ProcessData:
    process_id = 0
    project_id = 0
    task_id = 0
    start_time = 0
    end_time = 0
    user_id = 0
    weekend_id = 0

An instance of this class needs to be pickled into a data string by doing data_string = pickle.dumps(ProcessData()) and unpickled by doing data_variable = pickle.loads(data) where data is what is received.

So let us consider a case where the client creates an object of ProcessData and sends it to server. Here's what the client would look like. Here's a minimal example.

Client

import socket, pickle

class ProcessData:
    process_id = 0
    project_id = 0
    task_id = 0
    start_time = 0
    end_time = 0
    user_id = 0
    weekend_id = 0


HOST = 'localhost'
PORT = 50007
# Create a socket connection.
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))

# Create an instance of ProcessData() to send to server.
variable = ProcessData()
# Pickle the object and send it to the server
data_string = pickle.dumps(variable)
s.send(data_string)

s.close()
print 'Data Sent to Server'

Now your server which receives this data looks as follows.

Server

import socket, pickle

class ProcessData:
    process_id = 0
    project_id = 0
    task_id = 0
    start_time = 0
    end_time = 0
    user_id = 0
    weekend_id = 0


HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print 'Connected by', addr

data = conn.recv(4096)
data_variable = pickle.loads(data)
conn.close()
print data_variable
# Access the information by doing data_variable.process_id or data_variable.task_id etc..,
print 'Data received from client'

Running the server first creates a bind on the port and then running the client makes the data transfer via the socket. You could also look at this answer.

Schlessel answered 20/11, 2017 at 16:8 Comment(11)
I have run this code many times. This doesnt work. Have you run it yourself and checked?Dup
What do you mean you've run it and it doesn't work. How are you running the individual files? Can you open two terminals, copy the code above into the respective files server.py and client.py. Run the server in one terminal and client in another. You'll get a <__main__.ProcessData instance at 0x10572edd0> as a response which is an instance of ProcessData class you've sent from client to server.Schlessel
@Dup This example works perfectly, shows and explains what you need to do PERFECTLY. This solves the question you asked, please mark it as accepted.Chic
It works!!!! Awesome Sudheesh. Thank you. Thats really great. I will mark it as answer.Dup
@SudheeshSinganamalla apologies all my comments was badly stuck with this. I am new to Python. Thanks for the help buddy. I appreciate it.Dup
@Bilkokuya Thank you very much. @Dup Glad it helped. However, I would like to tell you something. You mentioned in your comments above Instead responding to my last comment you could have answered the question or could have said that it is out of my knowledge. Please understand we are not doing you a favour and you shouldn't be speaking to anyone in the stackoverflow community this way. All the best with your work.Schlessel
@SudheeshSinganamalla will keep that in mind. Thanks again :)Dup
@SudheeshSinganamalla my friend thats working fine with straight single server receipt. But when the data is large I need to concatenate the data. There it is failing. I believe data is not being formed properly. Please see my sent code how I am receiving the data on the server side. There it says UnpicklingError: invalid load key, '\xff'.Dup
The issue is because of the limit 4096 bytes. You need a loop to receive all the data and then join the 4096 byte chunks into the string you want. Here is an example. You can easily search for these on stackoverflow. Please don't expect us to write everything.Schlessel
is there a way to send the client a python object at the start of the connection?Rescissory
Can this be done without the client having the class defined? It's not ideal having two identical definitions.Probationer
T
0

Shameless plug here, but a friend and I have recently released tlspyo, an open-source library whose purpose is to help you transfer python objects over network easily and in a secure fashion.

Transferring pickled objects via Internet sockets without using something like tlspyo is basically an open door for hackers, so don't do it.

With tlspyo, your code looks like this:

Server:

from tlspyo import Relay

if __name__ == "__main__":
    my_server = Relay(port=3000,password="<same strong password>")

    # (...)

Client 1:

from tlspyo import Endpoint

if __name__ == "__main__":
    client_1 = Endpoint(
        ip_server='<ip of your server>'
        port=3000,
        password="<same strong password>",
        groups="client 1")

    # send an object to client 2:
    my_object = "my object"  # doesn't have to be a string, of course
    client_1.broadcast(my_object, "client 2")

    # (...)

Client 2:

from tlspyo import Endpoint

if __name__ == "__main__":
    client_2 = Endpoint(
        ip_server='<ip of my Relay>'
        port=3000,
        password="<same strong password>",
        groups="client 2")

    # receive the object sent by client 1:
    my_object = client_2.receive_all(blocking=True)[0]


    # (...)

(You will need to setup TLS for this code to work, check out the documentation - or you can disable TLS using security=None, but if you are transferring over the Internet you don't want to do that.)

Tantalous answered 21/11, 2022 at 8:21 Comment(0)
P
-1

An option is to use JSON serialization.

However, Python objects are not serializable, so you have to map your class object into Dict first, using either function vars (preferred) or the built-in __dict__.

Adapting the answer from Sudheesh Singanamalla and based on this answer:

Client

import socket, json

class ProcessData:
    process_id = 0
    project_id = 0
    task_id = 0
    start_time = 0
    end_time = 0
    user_id = 0
    weekend_id = 0


HOST = 'localhost'
PORT = 50007
# Create a socket connection.
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))

# Create an instance of ProcessData() to send to server.
variable = ProcessData()

# Map your object into dict
data_as_dict = vars(variable)

# Serialize your dict object
data_string = json.dumps(data_as_dict)

# Send this encoded object
s.send(data_string.encode(encoding="utf-8"))

s.close()
print 'Data Sent to Server'

Server

import socket, json

class ProcessData:
    process_id = 0
    project_id = 0
    task_id = 0
    start_time = 0
    end_time = 0
    user_id = 0
    weekend_id = 0


HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print 'Connected by', addr

data_encoded = conn.recv(4096)
data_string = data_encoded.decode(encoding="utf-8")

data_variable = json.loads(data_string)
# data_variable is a dict representing your sent object

conn.close()
print 'Data received from client'

Warning

One important point is that dict mapping of an object instance does not map class variable, only instance variable. See this answer for more information. Example:

class ProcessData:
    # class variables
    process_id = 0
    project_id = 1

    def __init__(self):
        # instance variables
        self.task_id = 2
        self.start_time = 3

obj = ProcessData()
dict_obj = vars(obj)

print(dict_obj)
# outputs: {'task_id': 2, 'start_time': 3}

# To access class variables:
dict_class_variables = vars(ProcessData)

print(dict_class_variables['process_id'])
# outputs: 0
Politesse answered 21/4, 2021 at 13:49 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.