Linux driver DMA transfer to a PCIe card with PC as master

Question

I am working on a DMA routine to transfer data from PC to a FPGA on a PCIe card. I read DMA-API.txt and LDD3 ch. 15 for details. However, I could not figure out how to do a DMA transfer from PC to a consistent block of iomem on the PCIe card. The dad sample for PCI in LDD3 maps a buffer and then tells the card to do the DMA transfer, but I need the PC to do this.

What I already found out:

1. Request bus master

   pci_set_master(pdev);
   

2. Set the DMA mask

   if (dma_set_mask(&(pdev->dev), DMA_BIT_MASK(32))) {
       dev_err(&pdev->dev,"No suitable DMA available.\n");
       goto cleanup;
   }
   

3. Request a DMA channel

   if (request_dma(dmachannel, DRIVER_NAME)) {
       dev_err(&pdev->dev,"Could not reserve DMA channel %d.\n", dmachannel);
       goto cleanup;
   }
   

4. Map a buffer for DMA transfer

   dma_handle = pci_map_single(pci_dev, buffer, count, DMA_TO_DEVICE);
   

Question:

What do I have to do in order to let the PC perform the DMA transfer instead of the card?

Thank your for your help!

---

First of all thank you for your replies. Maybe I should put my questions more precisely:

1. In my understanding the PC has to have a DMA controller. How do I access this DMA controller to start a transfer to a memory mapped IO region in the PCIe card?
2. Our specification demands that the PC's DMA controller initiates the transfer. However, I could only find examples where the device would do the DMA job (DMA_mapping.txt, LDD3 ch.15). Is there a reason, why nobody uses the PC's DMA controller (It still has DMA channels though)? Would it be better to request a specification change for our project?

Thanks for your patience.

Chick

"transfer data from PC" has meaning for external peripherals, e.g. "from PC to a printer". For internal transfers it makes no sense. You probably mean main memory. "let the PC perform the DMA transfer" also makes no sense. You probably mean the DMA Controller. Answer: setup the DMAC instead of the PCI bus master. – Odeliaodelinda

Answer 1

Look up DMA_mapping.txt. There's a long section in there that tells you how to set the direction ('DMA direction', line 408).

EDIT

Ok, since you edited your question... your specification is wrong. You could set up the system DMA controller, but it would be pointless, because it's too slow, as I said in the comments. Read this thread.

You must change your FPGA to support bus mastering. I do this for a living - contact me off-thread if you want to sub-contract.

Farflung

Yes, I read that. The code there tells how to set up a buffer for DMA transfer. Same construct (pci_map/pci_unmap) is used in LDD3 ch. 15, but there are also instructions to activate the DMA controller on the PCI card. Therefore I suppose this only marks the buffer as "DMA ready" and let the device to the DMA transfer. The direction only tells that the DMA controller should read or write the buffer (or both). It doesn' tell me (or at least I didn't get that) how to activate the PCs DMA controller. – Chick

You don't "activate the PCs DMA controller". There are 2 ways to do DMA on PCIe: (1) "system" DMA, and (2) "Bus master" DMA. (1) is history; it's not an option. The "system" DMA controller is still on your PC, but it's in the wrong place and is too slow. Your FPGA must instead become a bus master and burst the required data either in or out. So, you set up the DMA direction as per DMA_mapping.txt, and then write to a register on your card to initiate a bus master burst read (PC out to FPGA). Check your FPGA vendor app notes; start with XAPP1052. Note also that LDD3 is pretty useless. – Farflung

Not enough space above... who coded your FPGA? Do they understand bus master operation? Do you have a control bit to enable bus mastering? If not, then your FPGA cannot "DMA" in either direction. – Farflung

The specification was changed. FPGA does the DMA job now. Problem solved. Thank you! – Chick

@Farflung The link you refer is dead. Could you please briefly explain why DMA on the PC side is slow to push data from CPU side memory to the FPGA memory? And Why is DMA on FPGA side acts as a bus master faster at this kind of transactions? – Timely

Answer 2

What you are talking about is not really a DMA. The DMA is when your device is accessing memory and the CPU itself is not involved (with an exception of PC's memory controller, which is usually embedded into the PC's CPU these days). Not all devices can do it, and if you are using FPGA, then you surely need some sort of DMA controller in your design (i.e. Expresso DMA Core or alike). In your case, you just have to write to the mapped memory region (i.e. one that you obtain with ioremap_nocache) using iowrite calls (i.e. iowrite32) followed by write memory barriers wmb(). What I/O bar and address you have to write to entirely depends on your device.

Hope it helps. Good Luck!

Tressatressia

"What you are talking about is not really a DMA" ... "then you surely need some sort of DMA controller in your design" -- you seem to be referring to a (PCI/PCIe) board performing bus mastering. A peripheral device (designed with the additional DMA control/handshake signals) can utilize DMA (rather than programmed I/O) by using the system's DMA controller (which was included in the IBM PC design). – Odeliaodelinda

@Odeliaodelinda yes you are right, bus mastering it is. I am sometimes very bad with the correct symbology. I don't think today's commodity PCs have those DMA controllers, or do they? – Tressatressia

PCs have to have a DMAC to be "IBM PC compatible". A DMA controller is not that expensive. If you can put a processor that is clocked >350 times faster than the original i8088 (and include a numeric coprocessor) and have more than 1000 times more memory, then a DMAC is affordable in a "commodity PC". – Odeliaodelinda

@VladLazarenko: Yes, I can iowrite to the FPGA's memory block. Tested it and it works. Next step would be to have the PC's DMA controller transfer data to this memory region, but all I found was examples on reserving a memory buffer and have the card's DMA controller do the job (including DMA_mapping.txt, LDD3 ch. 15). The question is, how to activate the PC's DMA controller since specification document says, that DMA has to come from the PC's side. – Chick

The name "DMA" indicates your CPU is not doing the active transfer. That immediately implies that something else has to do it for you. There is a "DMA controller" in PCs but as far as I know they work on ISA speeds and are not much use for PCI-e. You're going to need a DMA controller that does something useful for your target device's speed; typically you just add a DMA controller / busmaster to your FPGA design and use that. Alternatively, use the opposite - make the CPU do the active copy. – Badenpowell

When talking about "DMA" in PCI context, does it refer to DMA in the cpu/soc side (or to the external device) ? – Cohabit