I need faster floating point math for .NET C# (for multiplying and dividing big arrays)

Question

I need fastest possible way to multiply and divide big arrays of data.

I've read this (wrote by Ben Voigt here):

.NET doesn't use MMX or SSE or AVX, as of the current version

(...)

modern C++ compilers not only allow SIMD usage, but can auto-vectorize ordinary-looking code into SIMD instructions

.

I think I need:

• SSE support with vector operations (for multiplying 4 floats at once)

• multithreading support (solution/library that will not collide somehow with C# threading)

Is there any library/dll that i can use?

Edit: any alternatives for Octave? I neeed only 2 operations: divide, multiply. If I understood correctly what is Octave and how it works - I will need to parse output... It cant be fast...

---

According to "what have you tried" and "why you don't want to use simple for":

I need this for neural network training. Network (now) has more than 50 inputs, many neurons (each 50+ weights). Learning data contains 100.000+ rows, 50+ fields each. Each neuron input needs few (at least 5) multiply operations.

I have no idea how many learning epochs i need, but i tried to benchmark only multiply operations and i got result: about 16 secs per epoch on my Intel Core Duo T2500 2.0GHz CPU.

Of course i can buy faster computer, but new computer is worth more than few hours of my work, so I hope it looks logical...

Distributive

Are you sure you need SSE? Copying data into unmanaged memory or pinning can often times offset the performance benefit of using it. – Micrococcus

Is this really 50-dimensional data or just a matrix of size 50? For true 50 dimensional data you'd have a minimum of 2^50 values. What are you actually doing? – Vaticide

I don't know if this will do what you need but it may be able to help I herd a pod cast from the lead developer and stated it was as perfomant as native implementations. msdn.microsoft.com/en-us/library/ff524509(v=vs.93).aspx – Zebulen

@Vaticide this is 50-dimensional vector (one array with 50 elements), sorry for mistake – Distributive

@Micrococcus yes, i think need SSE, my data will be processed multiple times after moving to unmanaged memory. – Distributive

Maybe not what you want to hear, but have you tried in c# first before deciding it's too slow? – Ethelethelbert

Answer 1

When I saw this question, I searched for ways to use the GOTO BLAS libraries in C#. The GOTO libraries (named after the author, not the evil programming keyword) are widely considered the fastest CPU-based linear algebra libraries because they are written by a talented coder who tunes the library in assembly language for each specific CPU architecture (Opteron, Xeon, etc.)

It turns out that Math.NET Numerics is probably what you want.

From MSDN description:

Math.NET Numerics aims to be the standard open-source math library for the .NET Framework. It provides the methods and algorithms for numerical computations in science, engineering, and everyday use. The functionality covered by Math.NET Numerics includes special functions, linear algebra, probability models, statistics, random numbers, interpolation, and integral transforms (FFT). Math.NET Numerics provides a fully managed implementation that runs on .NET 4.0, Silverlight 4, and Mono (but can be compiled for other platforms). It also provides a parallelized managed implementation and supports optimization using native BLAS/LAPACK libraries (GotoBLAS, Intel MKL, and AMD ACML).

Tenantry

Looks interesting. Thanks. – Distributive

Under the hood this library appears to be wrapping the System.Math lib at least for things like the Trig functions. So if you are expecting speed ups it might not give you much of a change. – Elbow

Answer 2

Assuming your vectors are gigantic N-dimensional arrays/vectors

If I was using a Slow language like C# and wanted to multiply and divide huge arrays and needed to do it as quickly as possible utilizing all processors, I would have C# interface with GNU Octave. GNU octave is a vectorized language, so matrix multiplications use an order of magnitude less resources than doing nested for loops.

So you would define some custom scripts in GNU Octave to say multiply two 50 dimensional arrays together, then you have C# invoke octave and pass it the parameters. Then have C# gather the results.

GNU Octave was designed to utilize every trick in the book to make matrix calculations use as little resources as possible and complete as quickly as possible.

http://en.wikibooks.org/wiki/Octave_Programming_Tutorial/Getting_started

Someone else has asked how to interface C# with Octave:

Interfacing octave with C#

Edit: Doing all this extra work does not make sense unless calculations are taking more than say 10 minutes and it is super important to decrease processing time.

Selfsatisfaction

I was answering your original question, assuming you had a structure with 50 dimensions which is mind bogglingly big. Had I known you are trying to multiply/divide arrays of size 50, I would say use a C# for loop. – Selfsatisfaction

FYI, Kamil, calling a vector "50 dimensional" is not the same thing as saying an array or vector has 50 items in it. A 1 dimensional vector is a List, a 2 dimensional vector is a list of lists, a 3 dimensional vector is a list of list of lists. A 50 dimensional vector is possible, but a difficult concept to comprehend. – Selfsatisfaction

I know what you mean Eric. I messed up with nomenclature. Sorry for that. – Distributive

See my question update. I added information why i need fast math. – Distributive

Answer 3

I found something like this: Microsoft Research Accellerator v2

Accelerator is a high-level data parallel library which uses parallel processors such as the GPU or multicore CPU to accelerate execution.

Interesting features (paste from M$ site):

• Execution on multicore CPUs, both 32 and 64 bit, in addition to DX9 GPUs and CUDA.
• Ability to execute on multiple devices within a single Accelerator instance

Unfortunatelly I can't use it, its non-free for commercial use and I don't even want to ask them how much it cost..., probably too expensive for me.

Distributive

Answer 4

If you can afford to run it with Mono, the open-source alternate runtime for .NET, you can use Mono.Simd (http://docs.go-mono.com/index.aspx?link=N:Mono.Simd), which makes SIMD available around structs, but only in Mono, where the JIT inlines the method calls to SIMD operations.

Wilterdink

I dont want to change language or runtime. Application is quite big and almost completed. Rewriting will take too much time, and if I decide some day to do this - i'll rewrite it to C++ and i will use assembler (im familiar with "small CPU" ASM). – Distributive

You don't have to change the language, it is plain C#, you need only to pack your data in the structs to have operations done in parallel when you call the proper methods (which are inlined to SSE calls). But if you don't wan't to use the Mono runtime, alas, you won't have the benefits from SIMD. – Wilterdink

Can I write only math DLL in Mono, then use it as library in .NET? – Distributive