Why is EDI still used, and how to deal with it?

Question

Why is this archaic format still used in the face of easier-to-use technologies? Does it provide some benefit that I'm not seeing? It seems that a large amount of vendors still provide data only in this format, instead of something more manageable and easier to use such as XML; at the least it would make sense to me to offer both formats.

Also, what are some good ways to deal with and utilize EDI when you have no other choice but to use it? Something like BizTalk is out of the question as it's far too expensive. Are there any free/open source applications that make EDI easier to work with?

Duckling

Subjective and Argumentative: See 'cryptic', 'zero sense to me', other systems are 'more effecient', 'headache to parse', 'archaic' 'easier to use' 'read chinese'. – Imbalance

@Gortok: Yes, he is being subjective, but there is a question hidden inside the rant: What are the pros and cons of the EDI format? – Thermoluminescence

EDI stands for Electronic Data Interchange, not Human Data Interchange. Its not meant to be human readable. Its compact and well defined. I've converted 7MB EDI file to XML, and they ended up being 45MB+. The conversion size is not linear. Larger EDI files could get huge. – Aoristic

@Aoristic Same could be achieved for XML with some standard compression algorithm that no one needs to see. – Rissa

same reason why we use QWERTY keyboards – Palacio

Answer 1

EDI is not that hard to understand once you familiarize yourself with the delimiters it uses. You might ask yourself as well why anyone would still be using CSV or tab-delimited data.

The answer is probably that those formats are "domain specific languages" defined by committee and standardized in a certain industry, and that a lot of money has already been invested in supporting those formats. Where's the business case to throw that all out again?

Ninnyhammer

I've found that EDI is easy enough to read in. The difficult part is writing EDI documents that other systems don't complain about. It seems that even if you are following the specs, they still complain. – Bernettabernette

In fact, I've created a DSL to map EDI transaction sets (both ways). It was almost trivial. – Tongs

@Kibbee: Sounds like developing web sites for Internet Explorer! – Psalterium

Answer 2

One word, Inertia. Developing the EDI formats by committee between various companys and organisations with different agendas was a nightmare (sad to say I have been there).

Asking them to abandon these with yet another round of committees agreeing web service API standards is going to take even longer, how do you sell the idea of replacing one electronic format with another to a non-technical board? What possible busness advantage does it give them. Originally the benefits of electronic exchange were clear but replace one with another is not. We're talking really big companies here.

Social

Exactly. Suppliers aren't software development companies. They don't see the value in switching to new systems. They don't have the inhouse expertise to do anything, and hiring somebody would be extremely expensive. – Bernettabernette

not really "inertia"; it's due to the "network effect" en.wikipedia.org/wiki/Network_effect – Palacio

Answer 3

You may be interested in the following project:

http://bots.sourceforge.net/en/index.shtml Google code archive

Eldon

Answer 4

And switching to XML would give you what - a slightly easier to debug line format?

Generally you set it up and leave it, there isn't a lot of need to play with the raw EDI feed, certainly not enough to abandon the standard and start again.
There are lots of standards, like FAX that could be made more readable but no real pressing need to change them.

Custom

Have you ever review an EDI format, even something simple like an EDIFACT order? What tools are there to easily consume it and extend it? There might be some but we're talking several digits of currency to get what XML gives for free. – Social

+1 AnthonyWJones. Exactly my problem with EDI - it's ridiculously hard to parse whereas XML is easy to consume and extend. – Duckling

It's not a matter of slightly easier. EDI is orders of magnitude harder to do correctly than XML. – Bernettabernette

@AWJ I review them frequently (or used to, now it's filtered through for the most part). I have a much harder time finding the data embedded in XML cruft. With EDI I can insert a newline at the segment breaks and things more or less line up vertically within a screen-width. It's what you're used to. – Tongs

Answer 5

Because it's a formally established Standard (in fact a very large and comprehensive set of standards). And that's one of the claimed benefits of a standard - you won't need to change anything for a long time.

And to change it, it takes agreement between two or more (often thousands and thousands more) trading partners (including maybe all of your competitors) to agree.

EDI formats have much higher signal-to-noise ratios (because they were designed back when that was considered important.) Someone who knows and understands EDI will look at your XML and say "Where's the beef (data)?"

Very few developers write their own parsers. There are many good mappers available (and many legacy and enterprise apps come with them built in). So there's lots of relief available for your pain (including at least one Open Source app on SourceForge).

Tongs

Answer 6

A little information for all interested. EDI is basically a design by committee data exchange format that not only set out rules for data formatting (like XML), but also set out to define each document that could possibly ever be sent between 2 companies. So for any piece of data that could be exchanged between companies they came up with an exact definition of what was supposed to be in each of these documents. Of course, nobody could foresee every piece of data that 2 companies would want to exchange. So you end up with companies using fields that were defined for 1 thing, being used for some other piece of information.

What you ended up with, is an extremely convoluted data format, in which many people using it don't follow the standards, because they need to send custom information, which the standard doesn't account for. So in the end, you still need to talk to each company you want to deal with, and find out all the little idiosyncrasies of their implementation, just as you would have to do if you went to someone with a custom XML interface. Except that in the case of EDI, the format is hard to parse and even harder to write well, so you end up doing a whole bunch of work just to send a document, when doing the same kind of think with having a custom XML solution would have resulted in many times less problems.

Bernettabernette

But then you get trading partners who extend the XML formats for the same data and same purposes in wildly different ways, so the same consequences result. We all like our own hammer better than the others, but they all drive in the nails semi-crooked sometimes. – Tongs

My whole point is that you can't try to standardize the documents sent across the entire industry. Instead, everybody would just be better off if they just used XML, and defined their own DTD for others to adhere to, instead of trying to get everybody to agree on a single DTD, or data format. – Bernettabernette

Just because 100% of the message exchange requirements can't be standardized isn't a reason vote for a free for all approach. – Osric

Answer 7

"If it ain't broke, don't fix it."

Most of these organisations are processing vast amounts of data using EDI, and aren't about to change to something more modern without a compelling reason. And making things easy for third-party developers doesn't usually qualify, sad to say.

Denotation

Answer 8

IMHO there are several problems with EDIFACT.

1. It is not easy to parse or generate an Object model from it. This is probably not a big problem anymore as there are now good system around that do it for you e.g. smooks.org
2. It is not easy to read. You get used to but XML is a lot easier to read
3. Validation isn't that easy (compare that to validating XML)
4. There are far too many different versions and flavours, D95B, D96B, D00A, D00B etc.
5. But I think the biggest problem is that everyone is using the standards differently. They use the same 'format' but the fields are defined differently. We use EDIFACT to send and receive messages from Container Terminals and they all have slight differences. They would e.g. all use a D95B CODECO but for some terminals a certain segment is mandatory while for another it is optional or even not allowed to be there. Then you have segments that are used the same but the content in it is different.

So to summarise it: It is a pain in the neck.

Ayeshaayin

Answer 9

EDI is a very compact format and is often used to keep bandwidth usage in data exchanges as small as possible. The German customs offices for example use it in their ATLAS system to exchange a very high volume of data every day.

It is hard to parse and hard to read, but if the size of the resulting data matters, it can be a good choice and is supported by most of the bigger business applications.

Thermoluminescence

Towards this point alot of valued added networks (VANs) still charge kilocharacter rates. The more characters the more it costs. – Chape

Answer 10

Legacy Support

Cartilage

Answer 11

EDI is prolific in many industries. It would be prohibitively expensive to replace an already-working technology with a newer one.

Consider this, Walmart uses EDI to communicate with its vendors, stores, distribution chain, etc. I'm guessing they deal with tenss of thousands of vendors. Every one of them has sunk thousands of dollars into EDI technology. If Walmart decided to switch over to XML, its a decision that affects thousands of companies, not just Walmart.

This is true for any EDI user. After all, it's a standard used between trading partners.

I agree, EDI is a pain to work with. But 'back in the day', that's all we had.

Rebirth

Care to recommend anything to ease the pain of dealing with it, then? My frustration is because my company switched vendors and this new one only offers data in EDI format (the old one offered it in text as well), so I'm scrambling to figure out a way to integrate it with our systems – Duckling

@WayneM I may be missing something, but can't you use XSLT to transform your XML into EDI? XML->XSLT->EDI seems much easier than EDI->CustomeParser->XML... again I may be over simplifying. – Eldon

I'm not using XML; I just need to extract the data - I was hoping the supplier was using XML and not EDI since XML is easier to parse the data I need. – Duckling

@WayneM, there are a few companies that offer EDI mapping software that will translate EDI docs into user-defined formats. We used to use Sterling Commerce's product, but honestly, it was a love/hate relationship. – Rebirth

Answer 12

Edifact is one of the best standards when it comes down to document interchange. Most problems come from tradingpartners sending non standardized documents.

Yes it's a bit odd format and is tedious to work with if you don't know the ins and outs but that goes for XML as well.

You really want XML over Edifact? Look at the bloated, hard to read XML standards peppol (pan-european public procurement online) is working on.

Yes it's working nice and dandy if you don't have any errors in the systems, troubleshooting edifacts is so much easier once you get used to the format than troubleshooting UBL documents.

You say you have $0.00 to use on the project? You really should look into the amount of manual work done in your company and the costsavings EDI can offer some cost benefit analysis can be mighty handy.

Skelp

Have a look at edidev, stylusstudio or liaison Liaison has a free program EDI Notepad which is quite handy if you're unfamiliar with edifact There's also quite a few java parsers out there if it's gotta be opensource Agreed that BizTalk is a bit prizy but so is must large ERP systems, look at the prize in that context :-) – Skelp

Answer 13

What types of information can be exchanged via EDI?

A variety of types of business information exchange is available via EDI including:

-•Booking information

-•Bill of Lading information

-•Invoicing

-•Electronic Funds Transfer

-•Arrival Notice Information

-•Shipment Status Information

How would choosing EDI benefit my company? -•It streamlines the communications process between you and APL

-•It eliminates the need to rekey data, thus eliminating errors and the need to recheck information

-•It eliminates paper handling and the need for document storage

-•It improves the turntime and the accuracy of your data

-•It eliminates the need for faxing

Noria

Answer 14

One solution, although it will cost you, is to go to a company like ADX, which has tools you can use to convert EDI formats to more pleasing formats like CSV. Depending on the volume and type of transactions you are doing, this can be both affordable and a lot less stressful. I've used their products in the past, and while they are a bit of work to set up, they do work quite well, and are very stable. Because of the history of EDI, you could probably find hundreds of other companies that offer similar services.

Bernettabernette

Unfortunately, my budget is $0.00. I'll keep it in mind, though, in case I need to use it. – Duckling

Are they paying you $0.00 to do the work? Because if they aren't, you really should argue that it could be completed with far less overall cost to just pay someone else for a solution. – Bernettabernette

Answer 15

EDI has been around since before XML. Apart from the fact that two parties can pre-negotiate the EDI format that works for them both you must also consider the part of the VAN (value added network.)

In some cases the VAN performs validation of the message, or even reads the message and performs actions on it, such as copying it to additional parties based on its content.

The only reason really to use EDI is because "that's the way it's always been done", and therefore there is a lot of existing infrastructure around to support it. Why switch to XML when there is no need? And how is to say XML wont be replaced by JSON which will then be replaced by something else?

Bouncer

JSON is for data, XML is for documents. That is an important distinction. – Medlock

Please explain what you mean. – Bouncer

Answer 16

Another reason is that being business messages such as order. invoices, credit notes etc there is a lot of financial worth in the transactions and they need to be secure but perhaps more importantly they need have end to end validation and verification as well as non repudiation.

For example i send you an order for 1/2 million Euros worth of goods, you send me the goods, then i "lose" the order information and tell you i am not paying. The combination of the standards and the VANS make this almost impossible or at least with so much of an audit trail that it the problems could be tracked. This is why the "Oh let use xml and the internet instead of EDIFACT and the VANS" tend to fail. As someone els answered, Inertia, but it is an inertia founded in a stable effective, secure, reliable and well understood system.

Doing it on the cheap is not always an option.

If it is any consolation when i first implemented EDI in '87 there was virtually no software around and so i got the Interbridge tables and wrote my own parser for the UK TRADACOMS standard using Cognos software on and HP Mini, and it worked fine. Assuming you are trading with other EDI partners the cost probably comes at the point of needing to use a VAN.

Tambac

Answer 17

I've used EDI (ANSI X12 and EDIFACT) in 2 projects about Maritime Transport Logistics and found them to be very useful since most Ocean Carriers and Trading Partners accept them as the standard way of communication between their different systems.

So EDI format is still used and will continue to be used since it's a stablished standard and thousand companies have developed systems around them, and replacing them is a really big deal.

Hett

Answer 18

I've had to use EDI as well and I agree. We used BizTalk to map it which worked well. Many system are built on EDI(well before XML).

Deannadeanne

Maybe I wouldn't have the problem if I was using BizTalk... but unfortunately the price for it is waaaay too much to even bother considering as a solution. – Duckling