Why resourceReader.GetResourceData return Data of type "ResourceTypeCode.Stream" that is offset by 4

Question

In my function GetAssemblyResourceStream (code below), I read resource from Dll using "assembly.GetManifestResourceStream" and "resourceReader.GetResourceData".

When I set my memory stream from the byte array of the resource, I have to include an offset of 4 bytes:

const int OFFSET = 4;
resStream = new MemoryStream(data, OFFSET, data.Length - OFFSET);

What is the reason of that offset? where does it came from?

Reference: Sample at end of MSDN ResourceReader Class

Also: I made a test app to better understand resources. That app show the problem I had with the offset. My little test app is available at Github (VS 2015)

Update 2015-10-05 10h28 Due to very low answers, I suspected a bug and/or undocumented behavior. I reported a bug at Connect.Microsoft.com and will see the result.

Update 2015-10-07 I removed the bug. I still think it is not well documented and/or could be considered as a bug but I highly suspect they will close my request without doing anything. I hope nobody will fall in the same problem I did.

Code:

   // ******************************************************************
    /// <summary>
    /// The path separator is '/'.  The path should not start with '/'.
    /// </summary>
    /// <param name="asm"></param>
    /// <param name="path"></param>
    /// <returns></returns>
    public static Stream GetAssemblyResourceStream(Assembly asm, string path)
    {
        // Just to be sure
        if (path[0] == '/')
        {
            path = path.Substring(1);
        }

        // Just to be sure
        if (path.IndexOf('\\') == -1)
        {
            path = path.Replace('\\', '/');
        }

        Stream resStream = null;

        string resName = asm.GetName().Name + ".g.resources"; // Ref: Thomas Levesque Answer at:
        // https://mcmap.net/q/721413/-enumerating-net-assembly-resources-at-runtime

        using (var stream = asm.GetManifestResourceStream(resName))
        {
            using (var resReader = new System.Resources.ResourceReader(stream))
            {
                string dataType = null;
                byte[] data = null;
                try
                {
                    resReader.GetResourceData(path.ToLower(), out dataType, out data);
                }
                catch (Exception ex)
                {
                    DebugPrintResources(resReader);
                }

                if (data != null)
                {
                    switch (dataType) // COde from 
                    {
                        // Handle internally serialized string data (ResourceTypeCode members).
                        case "ResourceTypeCode.String":
                            BinaryReader reader = new BinaryReader(new MemoryStream(data));
                            string binData = reader.ReadString();
                            Console.WriteLine("   Recreated Value: {0}", binData);
                            break;
                        case "ResourceTypeCode.Int32":
                            Console.WriteLine("   Recreated Value: {0}", BitConverter.ToInt32(data, 0));
                            break;
                        case "ResourceTypeCode.Boolean":
                            Console.WriteLine("   Recreated Value: {0}", BitConverter.ToBoolean(data, 0));
                            break;
                        // .jpeg image stored as a stream.
                        case "ResourceTypeCode.Stream":
                            ////const int OFFSET = 4;
                            ////int size = BitConverter.ToInt32(data, 0);
                            ////Bitmap value1 = new Bitmap(new MemoryStream(data, OFFSET, size));
                            ////Console.WriteLine("   Recreated Value: {0}", value1);

                            const int OFFSET = 4;
                            resStream = new MemoryStream(data, OFFSET, data.Length - OFFSET);

                            break;
                        // Our only other type is DateTimeTZI.
                        default:
                            ////// No point in deserializing data if the type is unavailable.
                            ////if (dataType.Contains("DateTimeTZI") && loaded)
                            ////{
                            ////    BinaryFormatter binFmt = new BinaryFormatter();
                            ////    object value2 = binFmt.Deserialize(new MemoryStream(data));
                            ////    Console.WriteLine("   Recreated Value: {0}", value2);
                            ////}
                            ////break;
                            break;
                    }

                    // resStream = new MemoryStream(resData);
                }
            }
        }

        return resStream;
    }

Nilsson

Answer 1

The 4 bytes at the start of the byte[] is the size of the data that follow the size. But it is totally useless because it is part of a byte[] and a byte[] size is already known. Moreover, the content of the stream is only one item where the 4 bytes offset could not serve to indicate the size of the first item in regards to following ones because there can't be any.

After reading ResourceReader.GetResourceData Method documentation : I tried both BinaryReader and BinaryFormatter without success. I will continue to read the content of the resource the same way I did earlier (bypass the size and convert directly to a stream with BitConverter).

Thanks to "Hey you" who gave me the idea to look in that direction.

Just for reference. This is my code but it could be not as accurate as it should be... It works for me but not deeply tested. Just to give a start.

// ******************************************************************
/// <summary>
/// Will load resource from any assembly that is part of the application.
/// It does not rely on Application which is specific to a (UI) frameowrk.
/// </summary>
/// <param name="uri"></param>
/// <param name="asm"></param>
/// <returns></returns>
public static Stream LoadResourceFromUri(Uri uri, Assembly asm = null)
{
    Stream stream = null;

    if (uri.Authority.StartsWith("application") && uri.Scheme == "pack")
    {
        string localPath = uri.GetComponents(UriComponents.Path, UriFormat.UriEscaped);

        int indexLocalPathWithoutAssembly = localPath.IndexOf(";component/");
        if (indexLocalPathWithoutAssembly == -1)
        {
            indexLocalPathWithoutAssembly = 0;
        }
        else
        {
            indexLocalPathWithoutAssembly += 11;
        }

        if (asm != null) // Take the provided assembly, do not check for the asm in the uri.
        {
            stream = GetAssemblyResourceStream(asm, localPath.Substring(indexLocalPathWithoutAssembly));
        }
        else
        {
            if (uri.Segments.Length > 1)
            {
                if (uri.Segments[0] == "/" && uri.Segments[1].EndsWith(";component/"))
                {
                    int index = uri.Segments[1].IndexOf(";");
                    if (index > 0)
                    {
                        string assemblyName = uri.Segments[1].Substring(0, index);

                        foreach (Assembly asmIter in AppDomain.CurrentDomain.GetAssemblies())
                        {
                            if (asmIter.GetName().Name == assemblyName)
                            {
                                stream = GetAssemblyResourceStream(asmIter, localPath.Substring(indexLocalPathWithoutAssembly));
                                break;
                            }
                        }
                    }
                }
            }

            if (stream == null)
            {
                asm = Assembly.GetCallingAssembly();
                stream = GetAssemblyResourceStream(asm, localPath.Substring(indexLocalPathWithoutAssembly));
            }
        }
    }
    return stream;
}

// ******************************************************************
/// <summary>
/// The path separator is '/'.  The path should not start with '/'.
/// </summary>
/// <param name="asm"></param>
/// <param name="path"></param>
/// <returns></returns>
public static Stream GetAssemblyResourceStream(Assembly asm, string path)
{
    // Just to be sure
    if (path[0] == '/')
    {
        path = path.Substring(1);
    }

    // Just to be sure
    if (path.IndexOf('\\') == -1)
    {
        path = path.Replace('\\', '/');
    }

    Stream resStream = null;

    string resName = asm.GetName().Name + ".g.resources"; // Ref: Thomas Levesque Answer at:
    // https://mcmap.net/q/721413/-enumerating-net-assembly-resources-at-runtime

    using (var stream = asm.GetManifestResourceStream(resName))
    {
        using (var resReader = new System.Resources.ResourceReader(stream))
        {
            string dataType = null;
            byte[] data = null;
            try
            {
                resReader.GetResourceData(path.ToLower(), out dataType, out data);
            }
            catch (Exception)
            {
                DebugPrintResources(resReader);
            }

            if (data != null)
            {
                switch (dataType) // COde from 
                {
                    // Handle internally serialized string data (ResourceTypeCode members).
                    case "ResourceTypeCode.String":
                        BinaryReader reader = new BinaryReader(new MemoryStream(data));
                        string binData = reader.ReadString();
                        Console.WriteLine("   Recreated Value: {0}", binData);
                        break;
                    case "ResourceTypeCode.Int32":
                        Console.WriteLine("   Recreated Value: {0}", BitConverter.ToInt32(data, 0));
                        break;
                    case "ResourceTypeCode.Boolean":
                        Console.WriteLine("   Recreated Value: {0}", BitConverter.ToBoolean(data, 0));
                        break;
                    // .jpeg image stored as a stream.
                    case "ResourceTypeCode.Stream":
                        ////const int OFFSET = 4;
                        ////int size = BitConverter.ToInt32(data, 0);
                        ////Bitmap value1 = new Bitmap(new MemoryStream(data, OFFSET, size));
                        ////Console.WriteLine("   Recreated Value: {0}", value1);

                        const int OFFSET = 4;
                        resStream = new MemoryStream(data, OFFSET, data.Length - OFFSET);

                        break;
                    // Our only other type is DateTimeTZI.
                    default:
                        ////// No point in deserializing data if the type is unavailable.
                        ////if (dataType.Contains("DateTimeTZI") && loaded)
                        ////{
                        ////    BinaryFormatter binFmt = new BinaryFormatter();
                        ////    object value2 = binFmt.Deserialize(new MemoryStream(data));
                        ////    Console.WriteLine("   Recreated Value: {0}", value2);
                        ////}
                        ////break;
                        break;
                }

                // resStream = new MemoryStream(resData);
            }
        }
    }

    return resStream;
}

Nilsson

I didn't find any documentation. It is only based on few tests where the 4 bytes was exactly the size of the data in the stream. Although the content were a large image or a small text, the 4 bytes (through BitConverter) were always the exact size. – Nilsson

Hey! Its me that should thank you for trying to help and drive me in the direction of investigating what look likes the value of the offset. Thanks !!! – Nilsson

Answer 2

I had the same problem and decided not to use GetResourceData method. I found other solution, example:

public static byte[] GetResource(string resourceName)
    {
        try
        {
            var asm = Assembly.LoadFrom("YOUR_ASSEMBLY");
            Stream str = asm.GetManifestResourceStream($@"YOUR_ASSEMBLY.Properties.Resources.resources"); 
            using (var reader = new ResourceReader(str))
            {
                var res = reader.GetEnumerator();
                while (res.MoveNext())
                {
                    if (res.Key.ToString() == resourceName)
                    {
                        if(res.Value is byte[])
                            return res.Value as byte[]; 
                        else if (res.Value is string)
                            return Encoding.UTF8.GetBytes(res.Value as string); 
// TODO other types

                    }
                }
                return null;
            }
        }
        catch (Exception ex)
        {
            // message
            return null;
        }
    }

Zackaryzacks

Thank you. I do not have my code to test it so I cannot assume it works fine for me. I guess it should be fine. So I did not mark it as the answer but I vote it up. Thank you... Other users will surely vote it up if they thinks it is a better solution. – Nilsson

Answer 3

To understand data offset it's important to understand the fundamentals of data streams:
What is a byte stream actually? and some of their uses: Transmission Control Protocol

The data offset is the number of bytes in the stream reserved for values other than the actual data. They are byte values that give format to the data.

So the offset needs to be included so those 4 bytes are not read as data. It's an instruction to the compiler of how to read the bytes.

In the case of uncompressed image data the offset is 4.

Pixels are packed into bytes and arranged as scan lines. Each scan line must end on a 4-byte boundary, so one, two, or three bytes of padding may follow each scan line.

All encoded data starts with a 0. Encoded bitmap data will start with bytes of code that is not to be read as the bitmap data, but rather explains what format the following data is to take.

Bytes of 04 AA indicates that there are 4 bytes of decoded data of value AA or AA AA AA AA If there is an odd number of decoded bytes, the run will start and end with padding, 00.

So 00 03 AA 00 will be decoded to AA AA AA

00 00 is a marker to show an end of scan line 00 01 is a marker for end of bitmap data

The run offset marker (aka delta or vector code) is 4 bytes. If it is present it is starts with the values:

00 02 XX YY followed by two more bytes, X and Y values.

The first two bytes 00 02 are instructions to indicate
-> that the following two bytes are the location, where the cursor should move to read the next line of data.

This run offset marker indicates the location in the bitmap where the next decoded run of pixels should be written. For example, a run offset marker value of 00 02 05 03 would indicate that the offset of the bitmap cursor should move five pixels down the scan line, three rows forward, and write out the next run. The cursor then continues writing decoded data from its new position moving forward.

Much of my information is sourced and quoted from:

Image and Data Compression at Microsoft Windows Bitmap File Format Summary

---

edit after feedback, it was not just images

Basically there can only be one of 19 data types. And for the Stream, the offset is added, as there is an assumption that the file type will likely have encoded markers, to describe the file type/format. As the Stream may be encoded, in which case that data cannot be read as it is, it needs to be decoded, which is what is provided in those offset bytes.

You could argue that xml is a string type, but if it is being detected as a Stream, there will be some embedded markers indicating it is not a string, but an encoded Stream.

In the case with the bmps, the reserved offset is 4, but the 4 bytes are not always filled, those bytes give instruction to the compiler of where to go next or how to interpret those next readable bytes, and when that instruction is complete (even if it's less than 4) the read will continue.

It's a warning, don't read these first bytes as data! the beginning 00 is a flag. As opposed to a 00 in an int type being read and converted to an int.

An xml file could have embedded markers, as does the bmps.

Chunk-based formats
In this kind of file structure, each piece of data is embedded in a container that somehow identifies the data. The container's scope can be identified by start- and end-markers of some kind, by an explicit length field somewhere, or by fixed requirements of the file format's definition.

This is the code behind the resource reader and I found it useful. If you look at this, you'll see there are only 19 data types.

reference source Microsoft resourcereader

I hope this has offered some clarity.

I don't believe this is a bug.

---

edit to add more explanation for clarity

The code type Stream is actually when the StreamWrapper Class is found.

See here line 565:

else if (type == typeof(StreamWrapper))
    return ResourceTypeCode.Stream; 

From the MSDN documentation.

This class wraps an IStream interface into a System.IO.Stream class. Since this code is for a classifier, there's no need to write to the provided stream since the pipeline only provides read-only streams to classifiers. Thus, the methods that modify the stream are not implemented.

hopefully this has answered your question

Glossitis

Thank you but the data is not a bitmap. The data is a stream. The content of the stream could be anything. In my case it was an xml file. Thanks for trying. You seems to be an expert in bitmap but I can't see how my problem could be related to bitmap. I grab code from Microsoft where they were extracting bitmap from the stream but it could be anything else. – Nilsson

It is possible that there is 1 or 2 invisible characters to tell which encoding (unicode etc). But I doubt that 4 would be the number. But mainly, there could be anything in the stream and I don't think the offset is related to the data it contains. I think it is a fixed offset that is always there regardless about the content of the stream. – Nilsson

It is possible that the 4 bytes is an identifier of what type of data is the resource. Ex: 0x001 : bitmap, 0x002: xmlFile. If it is the case, there should be a document that describe those 4 bytes enumeration or so. – Nilsson

I'm not sure... Let me some time to digest... Your reference to resourceReader could brings some lights. – Nilsson