Writing custom code for PowerPoint using leap motion?

Question

I am trying to make a gesture based PowerPoint slideshow. JavaScript would certainly not work here as I want my slideshow to work when that particular PowerPoint file (.ppt) is opened for windows.

I googled, posted a question on the Leap Motion forum, but in vain. My primary concerns are:

1. I am unsure about what to use here. Are there some SDKs for this purpose? In simple words, how do I write some custom code for a PowerPoint file on our system? Leap Motion apps can be developed with C#, C++, Python, JavaScript, Java, Objective-C

2. Integrating the code for Leap Motion device with the above code.

It's OK if you reply with any of the two points. I am sorry I couldn't post any code as I am absolutely new to PowerPoint-based programming for the system itself.

I am open to any language (C, C#, C++, Java, and Objective-C) that lets me accomplish my task. Sugesstions with some information/code would definitely help.

I am a PHP and JavaScript developer. I hope I am clear and not misunderstood.

Update:

I found LEAP Motion Controller Add-ins for Microsoft Office 2010/2013 (C#).

Kaete

feel free to correct the tagging if its for good, – Agave

Hi, it seems that here you can find useful information, source code and/or collaborators. – Phung

@razcor:thx,but is there any thing for mac..and is the tagging correct? – Agave

@razcor:How can I make the c++,python,java SO folks to see this question,i think i will need more than 5 tags here.. – Agave

I just retagged your post! I don't know if, at the moment, there are open source projects involving Mac Os X, Leap Motion and MS Office, where you can find information. – Phung

I think these tags are pretty popular on SO,let people come to know about the question,I would later change the tags accordingly..OS x,integration,python,SDK,Powerpoint are some other tags that could be handy – Agave

You could use C# to create Add-In for PowerPoint (e.g.: msdn.microsoft.com/en-us/library/vstudio/cc668192.aspx). You could also use Visual C++ to control and automate Microsoft Office (support.microsoft.com/kb/238972). – Verdha

Forgot to mention that in order to run .NET apps on Mac, you could use Mono (mono-project.com/Main_Page). – Verdha

Hey, not sure upon what you want to do. Actually VBScript can be used if you want to automate some work on powerpoint using its automation object. – Krasnoff

@codeomnitrix:i guess you know about leap motion?do you?I need to get started with programming for powerpoints..I hope I am clear now.. – Agave

To be used, the LEAP Motion Controller Add-ins would need to be installed locally on the machine running PowerPoint. Does that work for you? – Hienhieracosphinx

@JasonPlutext:the thing is i have never used an add-in.sdk before and never coded in c#.i have downloaded the SDK.But dont know how to go ahead? – Agave

Answer 1

The LEAP Motion Controller Add-ins for Microsoft Office which you found is likely to be the best way to integrate the controller with Powerpoint.

To get started with it, you'll need Visual Studio (seems like you'll need 2012). Microsoft has an overview page for Office Development in Visual Studio.

Follow the instructions on the Configuring a Computer to Develop Office Solutions page.

Download the Leap SDK for Windows, and unzip it, then in Visual Studio in the project GestureLib.NET4.0, add a reference to the LeapCSharp.NET4.0 dll

Make a trivial fix to GestureListener.cs (use IsEmpty at line 44).

After that you ought to be able to run the LEAP Motion Controller Add-in from within Visual Studio. It'll start Powerpoint when you do that.

"VSTO" is the name of the technology you are using here, so for more, Google 'VSTO add-in powerpoint'.

From the source code for the Add-In, it looks like you should see a single button on the ribbon in Powerpoint, for starting and stopping Leap.

Looking at ThisAddIn.cs, once started (by pressing the button on the ribbon), the controller ought to respond to left and right gestures, by moving to next/previous slides respectively:-

                if (direction.ToString() == "Right")
            {
                Application.ActivePresentation.SlideShowWindow.View.Next();
                LastGesture = DateTime.Now;
            }
            if (direction.ToString() == "Left")
            {
                Application.ActivePresentation.SlideShowWindow.View.Previous();
                LastGesture = DateTime.Now;
            }

GestureLib supports additional gestures, which you could make do something following that same pattern.

Hienhieracosphinx

I would like to have a chat with you..let me know when you are on stackoverflow? i found this code.msdn.microsoft.com/office/How-to-Automate-control-23cd2a8f – Agave

OK, well that one is automating Powerpoint. Com interop is a different approach to VSTO (not using VSTO, even though it says it does), and you could do things that way. See #1569193 and blogs.msdn.com/b/ericwhite/archive/2010/12/02/… – Hienhieracosphinx

:ok,i saw the code in add-in and i think that not much coding part is left,reason being they covered all the parts in add in...I just need to import all the files in my visual studio and wat else? – Agave

see my edits above; eventually you'll need to think about deployment. Google 'ClickOnce deployment' for one approach to this. – Hienhieracosphinx

ok,after importing these file to my project folder,i need to figure out the code for gesture detection and its deployment? – Agave

Nice solution! Just a side note: VSTO is only available for VS Professional or above (see en.wikipedia.org/wiki/Visual_Studio_Tools_for_Office). It will not work with the free Express editions nor with SharpDevelop. – Izzo

Answer 2

A very simple solution could look like this:

1. Build a basic C# application based on the LeapMotion API (you can use the C# example app from ther SDK). This application must be allowed to run in the background. It connects to the LeapMotion and waits for gestures. It provides a .NET remoting interface (see here for an easy example) and raises an event (see here for an example of events with remoting) for each interesting gesture. When you use the example app, just replace some of the SafeWriteLine(...) with the raising the appropriate event.
2. Build an office add-in with C# (just use NetOffice, it is free). This add-in starts the background app from step 1, connects with remoting and waits for events.
3. When the add-in gets an event, it triggers the according actions in PowerPoint (e.g. presentation.GotoSlide(presentation.Slides.Count)).

Izzo

hey,thx for the answer but if possible can you add some detailing to your answer?have you build a leap app before? – Agave

I am part of the developer program. But I have no public available leap motion app (yet). – Izzo

well i am currently learning c#,would this be possible in python also...i mean without any browser..So I dint understood most of the part that are mentioned in the answer..but if i need some help,would you if possible..by mail or chat?is your suggestion and jsonplutext's suggestion the same..he is recommending about VSTO.. – Agave

For the office plugin you will need DotNET, but you could try IronPython (ironpython.net) with NetOffice. For the LeapMotion part, just look at the python example (developer.leapmotion.com/documentation/Languages/Python/Guides/…). Instead of remoting, your Python program can export a WebService or you can use ZeroMq (zeromq.org). – Izzo

but if I dont want any internet to be used would that be possible on python or c# would be best.. – Agave

ZeroMQ with TCP on Windows is very fast. If you are concerned about security, just check that the connection comes from localhost or include a randomly created UUID in each request, stored in a local file accessible by the Office plugin and the LeapMotion app and refreshed on each restart of the app. Otherwise use communication (e.g. with a WebService) over a named pipe (example: https://mcmap.net/q/752244/-createnamedpipe-in-python). – Izzo

And btw. you may find the ZeroMQ samples in Python interesting: github.com/imatix/zguide/tree/master/examples/Python – Izzo

let us continue this discussion in chat – Agave

Answer 3

If your aim is just to interact with a PowerPoint presentation then a simple solution might be to use BetterTouchTool.

It's designed for OS X, it works with the Leap Motion Controller, it allows you to map gestures (captured by the LM Controller) to keyboard shortcuts and as long as it's configured correctly, it works whilst it's running in the background.

Essentially, it should be possible to map a swipe with X fingers to the right as a press on the right arrow key - which will advance the presentation to the next slide. And so on.

If, however, you are looking for a programmatic challenge/hoping to make this in to an app for Airspace then you could use something like Apache POI's Java API.

This is a lot more complicated solution, but it does have the benefit that it will run on OS X - and requires you to muck around with some code :).

It seems like the easiest solution would be to follow the example for exporting the slides in to images. You now have a collection of images - one for each PowerPoint slide. This gives you quite a lot of options, such as

• You can treat this data an an in-memory database (and a web service) and go back to your comfort zone of PHP and JS and use the JS API for the Leap Motion Controller to capture gestures to drive essentially an online picture gallery.
• You can develop a pure Java solution using the Java API to capture gestures from the Leap Motion Controller and display the data using Swing/JavaFX or in the browser using JSF.

Hermeneutics

would this app 'better touch tool' let me control my ppt slides,when they are opened and how that mapping will be done.I request you for a little detailed answer.Do we have a similar app for windows also?have you done leap program integration before.thx – Agave

I believe so. I don't own a Mac, so I'm not 100% certain, but a friend of mine has used it with his Leap Motion Controller and found it to to integrate well. It's free, so it's worth downloading it and experimenting with it briefly. Judging by the website it should be possible to set up the mappings to key presses and then have it run in the background. There isn't a port for Windows and it seems that there isn't likely to be one. In my free time I'm experimenting with Leap Motion gesture integration – Hermeneutics

Answer 4

PowerPoint and all office tools supports visual basic .net macros. Simply wrote a vb macro for your ppt file that recognizes the gesture and trigger the page change action. This is enough.

If you want, C# code of leap motion can be integrated in your vb.net macro

Kindergartner

thx for the answer but to be honest,can I get some more detailing about what you are saying? – Agave

PowerPoint slides can contain Visual Basic code: for example you can execute code in reply to an event (for ex. a click). – Kindergartner

chat.stackoverflow.com/rooms/36636/… can you join the chat? – Agave