VDK Platform Guide: Linux-Booting Keraunos PCIe Tile

Version: 1.0
Date: March 26, 2026
Author: System Architecture Team

1. Introduction

This guide documents the final validated Synopsys Virtualizer (VDK) platform that boots RISC-V Linux on the host chiplet, enumerates a PCIe Endpoint connected to the Keraunos PCIe Tile, and enables data transfer from the host to memory attached to the tile’s noc_n_initiator port.

The reference implementation is at /localdev/pdroy/ruchika_ws/Keraunos_PCIE_Tile.

1.1 What This Platform Demonstrates

  • Full Linux boot on a RISC-V virtual CPU (TT_Rocket_LT) via OpenSBI

  • PCIe link-up between a DesignWare Root Complex (host) and Endpoint (device)

  • Linux kernel PCIe enumeration using the snps,dw-pcie driver

  • BAR0 mapping and MMIO access from Linux userspace

  • End-to-end data transfer: Host → RC → EP → PCIE_TILE → noc_n_initiator → Target_Memory

2. Platform Architecture

2.1 Two-Chiplet Topology

The VDK instantiates two chiplet groups with a direct PCIe link:

graph TB subgraph host["Host_Chiplet"] direction TB H_CPU["TT_Rocket_LT<br/>RISC-V rv64imac<br/>Runs Linux"] H_SMC["SMC (CPU + Peripherals)"] H_DRAM["DRAM<br/>256 MB @ 0x80000000"] H_UART["UART @ 0xC000A000"] H_PLIC["PLIC @ 0xC4000000"] H_RC["PCIE_RC<br/>DesignWare PCIe 2.0<br/>Root Complex"] H_MAP["SharedMemoryMap"] H_CLK["CLK_GEN"] H_RST["RST_GEN"] H_CPU --> H_SMC H_SMC --> H_MAP H_MAP --> H_DRAM H_MAP --> H_UART H_MAP --> H_PLIC H_MAP -->|DBI 0x44000000<br/>4 MB| H_RC H_MAP -->|AXI 0x70000000<br/>256 MB| H_RC end subgraph device["Keraunos_PCIE_Chiplet"] direction TB D_CPU["TT_Rocket_LT<br/>RISC-V<br/>pcie_bringup firmware"] D_SMC["SMC_Configure"] D_EP["PCIe_EP<br/>DesignWare PCIe 2.0<br/>Endpoint"] D_TILE["PCIE_TILE<br/>Keraunos PCIe Tile"] D_MEM["Target_Memory<br/>16 MB @ 0x0"] D_MAP["SharedMemoryMap"] D_CLK["CLK_GEN"] D_RST["RST_GEN"] D_CPU --> D_SMC D_SMC --> D_MAP D_EP -->|BusMaster| D_TILE D_TILE -->|pcie_controller_initiator| D_EP D_TILE -->|noc_n_initiator @ 0x0| D_MAP D_TILE -->|smn_n_initiator @ 0x0| D_MAP D_MAP --> D_MEM end H_RC <-->|"PCIMem ↔ PCIMem_Slave<br/>Direct PCIe Link"| D_EP H_RC -.->|"msi_ctrl_int → irqS[11]"| H_SMC style H_CPU fill:#e3f2fd style H_RC fill:#ffcccc,stroke:#c00 style D_EP fill:#fff3e0 style D_TILE fill:#c8e6c9,stroke:#2e7d32 style D_MEM fill:#ffe6cc style H_DRAM fill:#e1ffe1

2.3 PCIE_TILE Interface Connections

The Keraunos PCIe Tile connects to the chiplet’s SharedMemoryMap and EP:

Tile Port

Direction

Connected To

Address / Offset

pcie_controller_target

Target (in)

EP BusMaster

Inbound TLPs from host

pcie_controller_initiator

Initiator (out)

EP AXI_Slave

Outbound TLPs to host

noc_n_initiator

Initiator (out)

SharedMemoryMap (left)

Start 0x0

noc_n_target

Target (in)

SharedMemoryMap (right)

0x44400000, 16 MB

smn_n_initiator

Initiator (out)

SharedMemoryMap (left)

Start 0x0

smn_n_target

Target (in)

SharedMemoryMap (right)

0x18000000, 8 MB

3. Memory Maps

3.1 Host_Chiplet CPU Address Space

Address

Size

Target

Description

0x02000000

64 KB

CLINT

Machine timer and software interrupts

0x44000000

4 MB

PCIE_RC DBI

RC configuration registers (DesignWare DBI)

0x44300000

128 KB

PCIE_RC ATU

iATU outbound/inbound (via DBI CS2 range)

0x70000000

256 MB

PCIE_RC AXI_Slave

PCIe outbound window

0x70000000–0x70FFFFFF

16 MB

— Config

Type 0/1 Configuration TLPs (iATU-translated)

0x71000000–0x7FFFFFFF

240 MB

— MEM

Memory Read/Write TLPs to EP BARs

0x80000000

256 MB

DRAM

Host main memory (OpenSBI + Linux + initramfs)

0xC000A000

256 B

UART

DW APB UART, 115.2 MHz clock, 7.2 Mbaud

0xC4000000

2 MB

PLIC

RISC-V PLIC, 64 interrupt sources

3.2 Keraunos_PCIE_Chiplet SharedMemoryMap Decode

Configured via VPCFG range_mappings: 0x44000000:0x00400000:s;0x18000000:0x00800000;0x44400000:0x01000000;0x0:0x1000000

Address

Size

Target

Purpose

0x00000000

16 MB

Target_Memory (MEM)

Data memory — host-accessible via EP BAR0

0x18000000

8 MB

PCIE_TILE smn_n_target

SMN access into tile registers

0x44000000

4 MB

PCIe_EP AXI_DBI

EP DBI configuration (:s = secure)

0x44400000

16 MB

PCIE_TILE noc_n_target

NoC access into tile subsystem

4. Linux Boot

4.1 Boot Flow

  1. Virtualizer loads images: fw_payload.elf on Host_Chiplet CPU, pcie_bringup.elf on Device CPU

  2. OpenSBI initializes M-mode on Host CPU at entry 0x80000000

  3. OpenSBI transitions to S-mode and jumps to the embedded Linux kernel

  4. Linux parses the compiled device tree (keraunos_host.dtb)

  5. Kernel initializes CLINT (timer), PLIC (interrupts), UART (console)

  6. The snps,dw-pcie driver probes the PCIe RC at DBI 0x44000000

  7. Driver programs iATU outbound windows and scans bus 1

  8. EP is enumerated: vendor/device ID read, BARs assigned

  9. Linux mounts initramfs and launches /init (BusyBox shell)

  10. The pcie_xfer application is available for PCIe data transfer

4.2 Device Tree Highlights

/ {
    compatible = "tt,smc-pcie-tile";
    cpus { cpu@0 { riscv,isa = "rv64imac"; mmu-type = "riscv,sv39"; }; };
    memory@80000000 { reg = <0x0 0x80000000 0x0 0x10000000>; };  /* 256 MB */

    soc {
        clint@2000000  { reg = <0x0 0x02000000 0x0 0x10000>; };
        plic@c4000000  { riscv,ndev = <64>; };
        uart@c000a000  { clock-frequency = <0x6DDD000>; };       /* 115.2 MHz */

        pcie@44000000 {
            compatible = "snps,dw-pcie";
            reg = <0x0 0x44000000 0x0 0x400000>,                 /* DBI */
                  <0x0 0x70000000 0x0 0x01000000>;               /* config */
            bus-range = <0x0 0x1>;
            ranges = <0x02000000 0x0 0x71000000
                      0x0 0x71000000  0x0 0x0F000000>;           /* 240 MB MEM */
            interrupts = <32>, <33>;                              /* MSI, INTx */
            num-lanes = <4>;
        };
    };

    chosen {
        bootargs = "console=hvc0 earlycon=sbi rdinit=/init
                    pci=realloc pci=assign-busses pci=noaer pcie_aspm=off";
    };
};

4.3 Key Boot Parameters

Parameter

Value

Purpose

console=hvc0

HVC console

Uses SBI console calls (fastest in VP)

earlycon=sbi

SBI earlycon

Early console via OpenSBI ecall

rdinit=/init

Initramfs init

Uses embedded initramfs (no disk)

pci=realloc

PCI realloc

Reallocate PCI resources (no BIOS pre-assignment)

pci=assign-busses

Assign buses

Force bus number assignment

pci=noaer

Disable AER

Advanced Error Reporting not needed in VP

pcie_aspm=off

Disable ASPM

Active State Power Management not modeled

4.4 Toolchain Requirements

Component

Toolchain

ISA

Notes

Linux kernel

ASC GCC 14 (riscv64-unknown-linux-gnu-)

rv64imac

CONFIG_FPU=n

Initramfs (BusyBox)

Musl (riscv64-unknown-linux-musl-)

rv64imac / lp64

Soft-float, static

OpenSBI

ASC GCC 14

rv64imac

Builds fw_payload.elf

pcie_bringup (device)

Bare-metal GCC

rv64imac

No OS

5. PCIe Enumeration and BAR Assignment

5.1 Enumeration Topology

graph LR subgraph bus0["PCI Bus 0"] RC["Root Complex<br/>Type 1 Header"] end subgraph bus1["PCI Bus 1"] EP["Endpoint<br/>Type 0 Header<br/>BAR0: 16 MB"] end RC -->|"Downstream Port"| EP style RC fill:#e3f2fd style EP fill:#fff3e0

Property

Value

RC Bus/Dev/Fn

00:00.0

EP Bus/Dev/Fn

01:00.0

EP BAR0 size

16 MB (BAR0_MASK = 0xFFFFFF)

EP BAR0 type

64-bit Memory, non-prefetchable

Config access

iATU-translated (no ECAM hardware)

Host MEM window

0x71000000–0x7FFFFFFF

5.2 iATU Configuration

The dw-pcie Linux driver programs iATU outbound windows:

  • Window 0 (Config): Maps 0x70000000 (16 MB) → Type 0/1 config TLPs to bus 1

  • Window 1 (MEM): Maps 0x71000000 (240 MB) → Memory TLPs targeting EP BARs

Linux assigns BAR0 an address within the 0x71000000–0x7FFFFFFF MEM window. The physical address is visible in sysfs at /sys/bus/pci/devices/0000:01:00.0/resource.

6. Data Transfer Application (pcie_xfer)

6.1 Purpose

pcie_xfer is a Linux userspace C application that maps EP BAR0 into process address space (via sysfs resource0 or /dev/mem) and performs MMIO read/write operations. Each operation traverses the full data path to Target_Memory on the device chiplet.

6.2 Data Path

Host CPU (MMIO store)
  → RC AXI_Slave (0x70000000 window)
    → iATU translates to Memory Write TLP
      → RC.PCIMem → PCIe link → EP.PCIMem_Slave
        → EP.BusMaster
          → PCIE_TILE.pcie_controller_target
            → Internal tile fabric
              → noc_n_initiator
                → SharedMemoryMap decode
                  → Target_Memory @ 0x0 (16 MB)

6.3 Supported Commands

Command

Syntax

Description

Write

write <offset> <value>

Single 32-bit write at BAR0 + offset

Read

read <offset>

Single 32-bit read at BAR0 + offset

Fill

fill <offset> <count> <val>

Fill count dwords with a constant value

Dump

dump <offset> <count>

Hex dump count dwords from BAR0

Pattern

pattern <offset> <count>

Write `0xA0000000

Burst

burst <offset> <count>

Timed burst write, reports throughput

Verify

verify <offset> <count> <val>

Verify memory matches expected value

Send

send <offset> <file>

Write binary file contents to BAR0

6.4 EP Auto-Detection

The application walks /sys/bus/pci/devices/ to find the first non-bridge device (class != 0x0604xx or 0x0600xx), reads BAR0 physical address and size from the sysfs resource file, and maps it via resource0 (falling back to /dev/mem if needed).

6.5 Building pcie_xfer

Cross-compile for the host initramfs with the musl toolchain:

riscv64-unknown-linux-musl-gcc -O2 -static -o pcie_xfer pcie_xfer.c

The static binary is included in the initramfs CPIO archive.

7. Bare-Metal Test (pcie_e2e_test)

An alternative bare-metal test (pcie_e2e_test.c) runs on the Host_Chiplet without Linux. It provides:

  • Trap-safe MMIO operations (catch and report access faults)

  • RC DBI register dump (Vendor/Device ID, Command/Status, BARs, Capabilities)

  • Port Logic register inspection

  • iATU programming and verification

  • End-to-end data transfer test through the PCIe link

  • Detailed topology verification (Bus 0 RC, Bus 1 EP)

This test is useful for pre-Linux validation of the PCIe link and tile connectivity.

8. VP Configuration Reference

8.1 Available Configurations

Name

Active Config Path

Description

default

vpconfigs/default/default.vpcfg

Full Linux with standard kernel, quantum 6000 ps

mini_riscv64_linux

vpconfigs/mini_riscv64_linux/mini_riscv64_linux.vpcfg

Mini Linux for fast boot, quantum 1000 ps

Set the active config in snps.vpproject → activeConfig attribute.

8.2 Critical VPCFG Parameters

Host_Chiplet:

Parameter

Value

Purpose

RST_GEN.active_at_start

true

Start host CPU immediately

PCIE_RC.SHARED_DBI_ENABLED

false

Separate DBI window

PCIE_RC.cc_pipe_clk_Override

250

PIPE clock 250 MHz

PCIE_RC.cc_*_Override

100

AXI/DBI/aux clocks 100 MHz

initial_image

fw_payload.elf (empty load addr)

OpenSBI + Linux

quantum_value

6000 (default) / 1000 (mini)

Temporal decoupling quantum

UART_CLK

115200000

115.2 MHz UART clock

Keraunos_PCIE_Chiplet:

Parameter

Value

Purpose

RST_GEN.active_at_start

true

Start device CPU immediately

PCIe_EP.SHARED_DBI_ENABLED

false

Separate DBI window

PCIe_EP.cc_pipe_clk_Override

250

PIPE clock 250 MHz

initial_image

pcie_bringup.elf

Bare-metal EP init firmware

SharedMemoryMap_intf.range_mappings

See Section 3.2

Chiplet address decode

8.3 Image Loading

Critical: The host initial_image uses the format {path/fw_payload.elf,,,image+symbols,} with empty load address fields. The Virtualizer applies ELF program header addresses (entry at 0x80000000). Never set load address to 0x0 — this breaks OpenSBI execution.

9. Sideband Signal Mapping

9.1 EP ↔ PCIE_TILE Sideband

VDK Connection Source

VDK Connection Sink

Description

PCIe_EP.edma_int / edma_int_*

PCIE_TILE.pcie_misc_int

DMA interrupts

PCIe_EP.pcie_parc_int

PCIE_TILE.pcie_ras_error

Parity/RAS error

PCIe_EP.lbc_cii_hv

PCIE_TILE.pcie_cii_hv

CII header valid

PCIe_EP.lbc_cii_hdr_type

PCIE_TILE.pcie_cii_hdr_type

CII header type

PCIe_EP.lbc_cii_hdr_addr

PCIE_TILE.pcie_cii_hdr_addr

CII header address

PCIe_EP.cfg_flr_pf_active_x[0]

PCIE_TILE.pcie_flr_request

Function Level Reset

9.2 Clocks and Resets

Source

Destination

Notes

Chiplet.Infra.CLK_GEN

PCIE_TILE, PCIe_EP

System clock

Chiplet.Infra.RST_GEN

PCIE_TILE, PCIe_EP

System reset

Host.Infra.CLK_GEN

PCIE_RC

Host clock

Host.Infra.RST_GEN

PCIE_RC

Host reset

9.3 MSI Interrupt Path

PCIE_RC.msi_ctrl_int → Host_Chiplet.SMC.irqS[11] → PLIC IRQ → Linux ISR

10. Troubleshooting

10.1 Host Crashes or Nonsense PC

  1. Verify fw_payload.elf uses ELF-header load addresses (not forced 0x0)

  2. Check readelf -l fw_payload.elf — VirtAddr should show 0x8000xxxx

  3. Ensure kernel is built with CONFIG_FPU=n for rv64imac CPU

  4. Verify initramfs binaries are musl-compiled for rv64imac (no FPU instructions)

10.2 PCIe EP Not Detected

  1. Verify pcie_bringup.elf is loaded on device CPU and runs to completion

  2. Check that device RST_GEN.active_at_start = true

  3. Confirm direct PCIe link wiring (RC.PCIMem ↔ EP.PCIMem_Slave)

  4. Verify DT bus-range = <0x0 0x1> and bootargs include pci=assign-busses

10.3 BAR0 Access Returns All-Ones

  1. EP may not have BAR0 properly sized — check pcie_bringup programs BAR0_MASK

  2. iATU outbound window may not cover the assigned BAR address

  3. Verify chiplet SharedMemoryMap decode includes 0x0:0x1000000 for Target_Memory

10.4 Data Mismatch on Read-Back

  1. Check that noc_n_initiator is connected to SharedMemoryMap at start 0x0

  2. Verify Target_Memory size (16 MB) covers the access offset

  3. Confirm byte-enable and data width compatibility across the path