Python: How to transfer varrying length arrays over a network connection

Question

I need to transfer an array of varying length in which each element is a tuple of two integers. As an example:

path = [(1,1),(1,2)]
path = [(1,1),(1,2),(2,2)]

I am trying to use pack and unpack, however, since the array is of varying length I don't know how to create a format such that both know the format. I was trying to turn it into a single string with delimiters, such as:

msg = 1&1~1&2~
sendMsg = pack("s",msg) or sendMsg = pack("s",str(msg))

on the receiving side:

path = unpack("s",msg)

but that just prints 1 in this case. I was also trying to send 4 integers as well, which send and receive fine, so long as I don't include the extra string representing the path.

sendMsg = pack("hhhh",p.direction[0],p.direction[1],p.id,p.health)

on the receive side:

x,y,id,health = unpack("hhhh",msg)

The first was for illustration as I was trying to send the format "hhhhs", but either way the path doesn't come through properly.

Thank-you for your help. I will also be looking at sending a 2D array of ints, but I can't seem to figure out how to send these more 'complex' structures across the network.

Thank-you for your help.

Motorbus

Answer 1

While you can use pack and unpack, I'd recommend using something like YAML or JSON to transfer your data.

• Pack and unpack can lead to difficult to debug errors and incompatibilities if you change your interface and have different versions trying to communicate with each other.
• Pickle can give security problems and the pickle format might change between Python versions.

JSON is included in the standard Python distribution since 2.6. For YAML there is PyYAML.

Venola

Answer 2

You want some sort of serialization protocol. twisted.spread provides one such (see the Banana specification or Perspective Broker documentation). JSON or protocol buffers would be more verbose examples.

See also Comparison of data serialization formats.

Coben

Answer 3

If you include message length as part of the message, then you will know how much data to read. So the entire string should be read across the network.

In any case, perhaps it would help if you posted some of the code you are using to send data across the network, or at least provided more of a description.

Poling

Answer 4

Pack and unpack are mandatory? If not, you could use JSON and YAML.

Don't use pickle because is not secure.

Upbow

As Mark Byers mentions, pickle isn't safe if you're using this on a network where you don't want whoever is sending the data to be able to run arbitrary code on the recipient's machine. nadiana.com/python-pickle-insecure – Coben

I wasn't aware of these possible "exploits". – Upbow

Answer 5

Take a look at xdrlib, it might help. It's part of the standard library, and:

The xdrlib module supports the External Data Representation Standard as described in RFC 1014, written by Sun Microsystems, Inc. June 1987. It supports most of the data types described in the RFC.

Biquadratic