If you have ever tried to pull sales pricing condition records from SAP S/4HANA into SAP Datasphere using a generic OData connection and hit a TSV_TNEW_PAGE_ALLOC_FAILED short dump, you are not alone. High-volume pricing tables are one of the most common sources of memory failures in OData-based extraction. This post documents a tested solution: pagination for the initial full load and filter-based delta extraction for ongoing refreshes.

The Problem

The S/4HANA OData service API_SLSPRICINGCONDITIONRECORD_SRV exposes sales pricing condition records across multiple entity sets. When the underlying tables hold millions of rows — which is typical in any production system with complex pricing — a single unbounded OData GET request forces the ABAP application server to allocate all result rows into memory at once. The runtime error TSV_TNEW_PAGE_ALLOC_FAILED is the ABAP kernel's way of saying it ran out of roll area before it could return the response.

The knock-on effect is that neither the initial load nor any subsequent refresh completes successfully, leaving the Datasphere local table empty or stale.

 

Solution Overview

The fix has two parts that address different phases of the data pipeline:

• Pagination in Datasphere for the initial full load — prevents S/4 from building one giant result set in memory.
• Filter-based delta extraction for daily refreshes — uses an S/4 OData feature to retrieve only records changed within a configurable time window.

Part 1: Initial Full Load with Pagination

Connection Settings

Create or reuse a generic OData connection in Datasphere pointing to the S/4 communication arrangement URL. Set the OData version to V2, which is what API_SLSPRICINGCONDITIONRECORD_SRV exposes.

Data Model

The pricing API splits condition records across entity sets by validity type (date-dependent, quantity-based, and so on). The recommended data model is a union of those validity entities, then joined to the main record entity. This mirrors the logical structure of the underlying ABAP condition tables and avoids cartesian joins.

 

 

Data Flow Configuration

Open the data flow and configure three areas:

1. Data Flow level: Dynamic Memory Allocation

Enable this setting. Pricing condition data is wide and joins amplify row width further. Without dynamic allocation, the Datasphere execution engine can exhaust its default memory ceiling before the run finishes.

1. Source node: Fetch Size and Batch Query

In the properties of the OData source node, enable Fetch Size and Batch Query. This activates server-side pagination. When the data flow starts, Datasphere first sends a $count request to learn the total row count, then issues sequential OData calls using $top and $skip to page through the result set in manageable chunks. Neither call ever asks S/4 to extract all rows simultaneously, which eliminates the memory pressure that triggered the short dump.

Example paging sequence emitted by Datasphere:

GET …/A_SlsPrcgConditionRecord/$count

GET …/A_SlsPrcgConditionRecord?$top=50000&$skip=0

GET …/A_SlsPrcgConditionRecord?$top=50000&$skip=50000

…

1. Target node: Update Records By Primary Key (UPSERT)

 Enable UPSERT on the target node. This makes reruns safe: if a run is interrupted and restarted, already-written records are updated rather than duplicated. It also makes the same flow reusable for delta loads.

Part 2: Delta Extraction for Daily Refreshes

The S/4HANA Feature: NumberOfDaysSinceCreation , NumberOfDaysSinceLastChange

API_SLSPRICINGCONDITIONRECORD_SRV exposes a virtual filter property called NumberOfDaysSinceCreation , NumberOfDaysSinceLastChange on the A_SlsPrcgConditionRecord entity. Setting it to 1 returns only records that were created or modified in the last calendar day, without any custom ABAP code on the S/4 side.

Delta OData request:

GET /sap/opu/odata/sap/API_SLSPRICINGCONDITIONRECORD_SRV/A_SlsPrcgConditionRecord?$filter=NumberOfDaysSinceLastChange eq 1

You can parameterise this value in the Datasphere data flow to control the lookback window. Set it to 2 or 3 on Mondays to catch weekend changes.

Delta Data Model

For the delta flow, union two sets of records:

• Records filtered by creation date (new records).
• Records filtered by last-change date (updated records).

Both branches expand to the validity entity. In the OData source node properties, set Depth to 2. This instructs Datasphere to issue $expand to the associated validity entity in the same OData call, avoiding a separate roundtrip.

The target is the same local table used by the full load. With UPSERT enabled, changed records overwrite existing rows and new records are inserted cleanly on every scheduled run.

Monitoring the Runs

After starting a data flow run, open the data flow in Data Builder and switch to the Monitoring view. Two tabs are available:

• Messages: execution log entries, warnings, and error details.
• Metrics: row counts processed per source and target node, memory consumed, and elapsed time per stage.

For the paginated initial load, watch the row counter in Metrics increment in batches. This confirms that $top/$skip cycling is working rather than a single unbounded fetch that would trigger the memory error again.

Summary

Four settings in Datasphere and one S/4 OData feature eliminate the memory crash and enable a sustainable incremental load pattern:

• Dynamic Memory Allocation on the data flow: prevents Datasphere-side memory exhaustion on wide, joined result sets.
• Fetch Size + Batch Query (pagination) on the OData source: breaks S/4 reads into bounded $top/$skip pages, preventing TSV_TNEW_PAGE_ALLOC_FAILED.
• UPSERT on the target: makes reruns and delta merges idempotent.
• NumberOfDaysSinceLastChange filter: offloads delta filtering to S/4 with zero custom code.

The same pattern applies to any high-volume OData entity in S/4HANA. If you are hitting memory errors on a different service, start with pagination and dynamic memory allocation. Those two settings alone resolve the majority of cases.

Remark

During the OData service execution in the S/4 backend, you may not only run out of memory, but also encounter processing timeouts, e.g. 10 mins. In this case, you may need to create an additional source entity with different filters in your data flow. Or even a different data flow.

Source link