linux pcie RC 框架

1. linux pcie rc framework

 

Following is a brief explanation of layers shown in the diagram:

• There are different drivers for the connected PCIe devices like pci_endpoint_test, tg-3, r8169, xhci-pci, ahci, etc. It could be vendor-specific like most of the ethernet cards (tg3, r8169) or class-specific like xhci-pci and ahci. Each of these drivers will also interact with it’s own domain-specific stack. For example, tg3 will interface with network stack, and xhci-pci will interface with USB stack.

• The PCI core layer scans the PCIe bus to identify and detect any PCIe devices. It also binds the driver from the layer above, for the PCIe device, based on vendorid, deviceid and class.

• The PCI BIOS layer handles resource management. For example, allocation of memory resources for BARs.

• The bottom-most layer consists of the PCIe platform drivers like pcie-cadence, pcie-designware, etc. pci-j721e and pci-dra7xx are TI’s wrappers over these drivers. They configure platform-specific controllers and perform actual register writes. 

另一种架构图：

• arch pcie driver：放一些和架构强相关的pcie的函数实现，对应arch/arm64/kernel/pci.c

• acpi pcie driver: apci扫描时所涉及到的pcie代码，包括host bridge的解析初始化，pcie bus的创建，ecam的映射等， 对应drivers/acpi/pci*.c

• pcie core driver: pcie的子系统代码，包括pcie的枚举流程，资源分配流程，中断流程等，主要对应drivers/pci/*.c

• pcie port bus driver: 是pcie port的四个service代码的整合， 四个service主要指的是pcie dpc/pme/hotplug/aer，对应的是drivers/pci/pcie/*

• pcie ep driver：是叶子节点的设备驱动，比如显卡，网卡，nvme等。

2. pci device driver

基于pci core实现pci ep deivce driver

rtl8168 pcie网卡设备驱动向上与网络协议栈对接

pci_endpoint_test会虚拟成一个设备，在应用层直接通过ioctl来控制

3. pci core

kernel\linux\linux-4.19.125\drivers\pci\pci.c

3.1 register driver

pci_register_driver

module_pci_driver -> pci_register_driver

3.2 Device Initialization Steps(do this in pci device driver probe)

1) Enable the device

pci_enable_device() Before touching any device registers, the driver needs to enable the PCI device by calling pci_enable_device()

pci_set_master() will enable DMA by setting the bus master bit in the PCI_COMMAND register.

pcim_enable_device()  Managed pci_enable_device()

2) Request MMIO/IOP resources

The device driver needs to call pci_request_region() to verify no other device is already using the same address resource.

Generic flavors of pci_request_region() are request_mem_region() (for MMIO ranges) and request_region() (for IO Port ranges).

3) Set the DMA mask size (for both coherent and streaming DMA)

Drivers for all PCI-X and PCIe compliant devices must call pci_set_dma_mask() as they are 64-bit DMA devices.

4) Allocate and initialize shared control data 

pci_allocate_coherent()

5) Access device configuration space (if needed)

6) Register IRQ handler (request_irq())

request_irq()

pci_alloc_irq_vectors()

MSI capability can be enabled by calling pci_alloc_irq_vectors() with the

PCI_IRQ_MSI and/or PCI_IRQ_MSIX flags before calling request_irq().

7) Initialize non-PCI (i.e. LAN/SCSI/etc parts of the chip)

8) Enable DMA/processing engines

3.3 How to access PCI config space

pci_(read|write)_config_(byte|word|dword)

pci_bus_(read|write)_config_(byte|word|dword)

pci_find_capability()

3.4 Other interesting functions

pci_set_power_state()        Set PCI Power Management state (0=D0 ... 3=D3)

pci_find_capability()        Find specified capability in device's capability list.

pci_resource_start()        Returns bus start address for a given PCI region

pci_resource_end()        Returns bus end address for a given PCI region

pci_resource_len()        Returns the byte length of a PCI region

pci_set_drvdata()        Set private driver data pointer for a pci_dev

pci_get_drvdata()        Return private driver data pointer for a pci_dev

pci_set_mwi()            Enable Memory-Write-Invalidate transactions.

pci_clear_mwi()            Disable Memory-Write-Invalidate transactions.

pcim_set_mwi            a device-managed pci_set_mwi()

3.5 访问配置空间

pci_bus_read_config_byte

pci_bus_read_config_word

pci_bus_read_config_dword

pci_bus_write_config_byte

pci_bus_write_config_word

pci_bus_write_config_dword

3.6 other

pci_ioremap_bar(): 设备驱动程序调用pci_ioremap_bar()将写入EP BAR的总线地址（保存在resource数组中）映射到系统内存的虚拟地址，之后软件就可以通过虚拟地址访问PCI设备的存储空间

pci_remap_iospace

devm_pci_remap_cfgspace

pcim_iomap_regions

4. pci bios

linux-4.19.125/arch/arm64/kernel/pci.c

linux-4.19.125/arch/arm64/kernel/bios32.c

pcibios只有使能了CONFIG_ACPI，才会被使用。

5. pci host controller driver

5.1 dw_pcie_host_init使用到的PCI API

dw_plat_pcie_probe -> dw_plat_add_pcie_port -> dw_pcie_host_init:

• devm_pci_alloc_host_bridge

• devm_of_pci_get_host_bridge_resources

• devm_request_pci_bus_resources

• devm_pci_remap_cfgspace -> pci_remap_cfgspace

• pci_scan_root_bus_bridge

• pci_bus_size_bridges

• pci_bus_assign_resources

• pci_bus_add_devices

以上这些函数均在linux-4.19.125/drivers/pci/*中实现

也可以直接调用pci_host_probe带代替调用pci_scan_root_bus_bridge & pci_bus_size_bridges & pci_bus_assign_resources & pci_bus_add_devices这些函数

pci_host_probe -> pci_scan_root_bus_bridge & pci_bus_size_bridges & pci_bus_assign_resources & pci_bus_add_devices

如linux-4.19.125\drivers\pci\controller\pcie-cadence-host.c

cdns_pcie_host_probe -> devm_pci_alloc_host_bridge & devm_pci_remap_cfg_resource &  cdns_pcie_host_init & pci_host_probe

5.2 init RC