How to process (extremely) large EDI files with Stedi

Large EDI files are common across many industries, including healthcare (such as 834 Benefit Enrollments), logistics (210 Motor Carrier Freight Details and Invoices), and retail (846 Inventory Advice). Unlike most other EDI solutions, Stedi has virtually no file size limitations and can process EDI files that are gigabytes in size.

However, large files that have been translated into JSON still pose significant challenges for downstream applications. The translation ratio of an EDI file to JSON is typically 1:10, which means that a large EDI file can easily produce a multi-gigabyte payload that is difficult for downstream applications to receive and ingest.

To solve this problem, we’re excited to introduce Fragments, a new feature in our Large File Processing module. Fragments allow you to automatically split processed transactions into smaller chunks for easier downstream ingestion.

Use fragments to split large transactions

Large files are often the result of transactions containing many repeated loops or segments. A healthcare provider may send an 834 Healthcare Benefit Enrollment file containing tens of thousands of individual members – each one in the INS segment – or a brand may send an 846 Inventory Inquiry/Advice file containing millions of SKUs – each one in the LIN segment.

With Fragments, you can use repeated segments like these to split the transaction. For example, when you enable fragments on the INS loop in an 834, Stedi emits batches of INS loops instead of a single giant JSON payload.

For each fragment batch, Stedi emits a fragment.processed.v2 event, which you can use to automatically send Destination webhooks to your API. Fragment processed events contain details about the original transaction as well as the actual fragment payload itself.

Configure fragments

You can configure fragments for any transaction in minutes without writing any code. Before you begin, create a partnership in Stedi between you and your trading partner.

Set up the Stedi guide

First, you need to enable fragments in the Stedi guide for the transaction.

Stedi guides are a machine-readable format for the trading partner EDI specifications you typically receive as PDF or CSV files. The Stedi Network has pre-built guides for hundreds of popular partners that you can import into your account and use in your integrations for free.

You can enable fragments on one repeated segment within each guide. To enable fragments, open the guide in your Stedi account, click the segment, toggle Set as fragment to ON, and click Publish changes.

Create a transaction setting

Next, you need to create an inbound transaction setting. Transaction settings define the EDI transactions you plan to exchange with your trading partner and the guide Stedi should use to validate data.

When creating the transaction setting, choose the guide you previously configured with fragments and then toggle Enable fragments to ON.

Send fragments to your API

Finally, you can automatically deliver fragments to your API. To do this, add an event binding for fragment.processed.v2 events to a Destination webhook.

You can also use the Get Fragment API to manually retrieve fragments for processed transactions as needed.

  curl --request GET \
    --url https://core.us.stedi.com/2023-08-01/transactions/{transactionId}/fragments/{fragmentIndex} \
    --header 'Authorization: Key {STEDI_API_KEY}'

Get started on Stedi

Stedi allows you to seamlessly process large EDI files of virtually any size. You can also use fragments to efficiently manage large transactions and reduce the development work required to integrate with new trading partners. Check out the Fragments documentation for complete details.

To get started building EDI integrations on Stedi, book a demo with our team.