Login/Register
Using Ruby, how can I confirm that an XML snippit is valid?
Asked Answered
Solved rubyxmlxmppxmpp4r
D

4

3

As some of you make know, I'm working on XMPP (Jabber) integration for the StackOverflow chat system, as an XMPP component written in Ruby using the xmpp4r package.

I'm struggling with one issue (well, many issues, but one issue at the moment :-) I am taking the JSON feed from the chat and extracting the HTML for the messages. I am using The Ruby TidyHTML bindings to convert the HTML from the JSON fed to XHTML, so that I can send it as an XMPP message -- since XMPP messages are just XML, converting the HTML to XHTMl should make it valid XML which I can just stick into the <message> stanza.

For most messages, it works great!

My Mind Is Blown

However for other messages, it completely chokes -- the XMPP server closes the stream and the script grinds to a halt. (And rchern and others in The Tavern get upset. Well, maybe not upset, but they laugh at me. This makes me sad!)

I am almost certain that what's happing is, for some reason or other, the messages are not valid XML, and so the XMPP server is closing the connection because it encounters a parse error in the XML stream from the Ruby component. Here's an example of one such message:

<message to='[email protected]/Token' type='groupchat' xmlns='jabber:client'><body>&lt;div class=&quot;onebox ob-message&quot;&gt;&lt;a class=&quot;roomname&quot; href=&quot;/transcript/message/263372#263372&quot;&gt;&lt;span title=&quot;2010-11-04 19:20:23Z&quot;&gt;1 hour ago&lt;/span&gt;&lt;/a&gt;, by &lt;span class=&quot;user-name&quot;&gt;Fosco&lt;/span&gt; &lt;br/&gt;&lt;div class=&quot;quote&quot;&gt;&lt;div class=&quot;room-mini&quot;&gt;&lt;div class=&quot;room-mini-header&quot;&gt;&lt;h3&gt;&lt;img class=&quot;small-site-logo&quot; title=&quot;Gaming&quot; alt=&quot;Gaming&quot; width=&quot;16&quot; height=&quot;16&quot; src=&quot;http://sstatic.net/gaming/img/favicon.ico&quot; /&gt;&amp;nbsp;&lt;span class=&quot;room-name&quot;&gt;&lt;a href=&quot;http://chat.stackexchange.com/rooms/28/minecraft-talk&quot;&gt;Minecraft Talk&lt;/a&gt;&lt;/span&gt;&lt;/h3&gt;&lt;div class=&quot;room-mini-description&quot;&gt;Everything Minecraft, including classic and survival mode&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;room-current-user-count&quot; title=&quot;current users&quot;&gt;9&lt;/div&gt;&lt;div class=&quot;mspark&quot; style=&quot;height:25px;width:205px&quot;&gt;
&lt;div class=&quot;mspbar&quot; style=&quot;width:8px;height:13px;left:0px;&quot;&gt;&lt;/div&gt;&lt;div class=&quot;mspbar&quot; style=&quot;width:8px;height:9px;left:8px;&quot;&gt;&lt;/div&gt;&lt;div class=&quot;mspbar&quot; style=&quot;width:8px;height:2px;left:16px;&quot;&gt;&lt;/div&gt;&lt;div class=&quot;mspbar&quot; style=&quot;width:8px;height:8px;left:24px;&quot;&gt;&lt;/div&gt;&lt;div class=&quot;mspbar&quot; style=&quot;width:8px;height:1px;left:32px;&quot;&gt;&lt;/div&gt;&lt;div class=&quot;mspbar&quot; style=&quot;width:8px;height:1px;left:56px;&quot;&gt;&lt;/div&gt;&lt;div class=&quot;mspbar&quot; style=&quot;width:8px;height:0px;left:64px;&quot;&gt;&lt;/div&gt;&lt;div class=&quot;mspbar&quot; style=&quot;width:8px;height:0px;left:88px;&quot;&gt;&lt;/div&gt;&lt;div class=&quot;mspbar&quot; style=&quot;width:8px;height:0px;left:96px;&quot;&gt;&lt;/div&gt;&lt;div class=&quot;mspbar&quot; style=&quot;width:8px;height:11px;left:104px;&quot;&gt;&lt;/div&gt;&lt;div class=&quot;mspbar&quot; style=&quot;width:8px;height:7px;left:112px;&quot;&gt;&lt;/div&gt;&lt;div class=&quot;mspbar&quot; style=&quot;width:8px;height:7px;left:120px;&quot;&gt;&lt;/div&gt;&lt;div class=&quot;mspbar&quot; style=&quot;width:8px;height:25px;left:128px;&quot;&gt;&lt;/div&gt;&lt;div class=&quot;mspbar&quot; style=&quot;width:8px;height:14px;left:136px;&quot;&gt;&lt;/div&gt;&lt;div class=&quot;mspbar&quot; style=&quot;width:8px;height:4px;left:144px;&quot;&gt;&lt;/div&gt;&lt;div class=&quot;mspbar&quot; style=&quot;width:8px;height:7px;left:152px;&quot;&gt;&lt;/div&gt;&lt;div class=&quot;mspbar&quot; style=&quot;width:8px;height:19px;left:160px;&quot;&gt;&lt;/div&gt;&lt;div class=&quot;mspbar&quot; style=&quot;width:8px;height:19px;left:168px;&quot;&gt;&lt;/div&gt;&lt;div class=&quot;mspbar&quot; style=&quot;width:8px;height:12px;left:176px;&quot;&gt;&lt;/div&gt;&lt;div class=&quot;mspbar&quot; style=&quot;width:8px;height:11px;left:184px;&quot;&gt;&lt;/div&gt;&lt;div class=&quot;mspbar now&quot; style=&quot;height:25px;left:154px;&quot;&gt;&lt;/div&gt;&lt;/div&gt;
&lt;div class=&quot;clear-both&quot;&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;</body><html xmlns='http://jabber.org/protocol/xhtml-im'><body xmlns='http://www.w3.org/1999/xhtml'><div class="onebox ob-message"><a class="roomname" href="/transcript/message/263372#263372"><span title="2010-11-04 19:20:23Z">1 hour ago</span></a>, by <span class="user-name">Fosco</span><br />
<div class="quote">
<div class="room-mini"><div class="room-mini-header">
<h3><img class="small-site-logo" title="Gaming" alt="Gaming" width="16" height="16" src="http://sstatic.net/gaming/img/favicon.ico" />&nbsp;<span class="room-name"><a href="http://chat.stackexchange.com/rooms/28/minecraft-talk">Minecraft Talk</a></span></h3>
<div class="room-mini-description">Everything Minecraft, including classic and survival mode</div>
</div>
<div class="room-current-user-count" title="current users">9</div>
<div class="mspark" style="height:25px;width:205px">
<div class="mspbar" style="width:8px;height:13px;left:0px;"></div>
<div class="mspbar" style="width:8px;height:9px;left:8px;"></div>
<div class="mspbar" style="width:8px;height:2px;left:16px;"></div>
<div class="mspbar" style="width:8px;height:8px;left:24px;"></div>
<div class="mspbar" style="width:8px;height:1px;left:32px;"></div>
<div class="mspbar" style="width:8px;height:1px;left:56px;"></div>
<div class="mspbar" style="width:8px;height:0px;left:64px;"></div>
<div class="mspbar" style="width:8px;height:0px;left:88px;"></div>
<div class="mspbar" style="width:8px;height:0px;left:96px;"></div>
<div class="mspbar" style="width:8px;height:11px;left:104px;"></div><div class="mspbar" style="width:8px;height:7px;left:112px;"></div><div class="mspbar" style="width:8px;height:7px;left:120px;"></div><div class="mspbar" style="width:8px;height:25px;left:128px;"></div><div class="mspbar" style="width:8px;height:14px;left:136px;"></div>
<div class="mspbar" style="width:8px;height:4px;left:144px;"></div>
<div class="mspbar" style="width:8px;height:7px;left:152px;"></div>
<div class="mspbar" style="width:8px;height:19px;left:160px;"></div>
<div class="mspbar" style="width:8px;height:19px;left:168px;"></div><div class="mspbar" style="width:8px;height:12px;left:176px;"></div>
<div class="mspbar" style="width:8px;height:11px;left:184px;"></div>
<div class="mspbar now" style="height:25px;left:154px;"></div>
</div>
<div class="clear-both"></div>
</div>
</div>
</div>
</body></html></message>

(This message happened to be a quote of a oneboxed link to a chat room)

Here was the error Ruby gave me:

IOError: stream closed
/usr/lib/ruby/1.8/xmpp4r/stream.rb:594:in `empty?'
/usr/lib/ruby/1.8/rexml/parsers/baseparser.rb:153:in `empty?'
/usr/lib/ruby/1.8/rexml/parsers/baseparser.rb:193:in `pull'
/usr/lib/ruby/1.8/rexml/parsers/sax2parser.rb:92:in `parse'
/usr/lib/ruby/1.8/xmpp4r/streamparser.rb:79:in `parse'
/usr/lib/ruby/1.8/xmpp4r/stream.rb:75:in `start'
/usr/lib/ruby/1.8/xmpp4r/stream.rb:72:in `initialize'
/usr/lib/ruby/1.8/xmpp4r/stream.rb:72:in `new'
/usr/lib/ruby/1.8/xmpp4r/stream.rb:72:in `start'
/usr/lib/ruby/1.8/xmpp4r/connection.rb:119:in `start'
/usr/lib/ruby/1.8/xmpp4r/component.rb:70:in `start'
/usr/lib/ruby/1.8/xmpp4r/connection.rb:77:in `connect'
/usr/lib/ruby/1.8/xmpp4r/component.rb:47:in `connect'
./classes/SOXMPP_Bridge.rb:20:in `initialize'
./soxmpp.rb:81:in `new'
./soxmpp.rb:81

Finally, my question!

Given that sending invalid XML to the XMPP server kicks me off, is there any way using Ruby I can validate (and, preferably, correct) the XML before sending it to the XMPP server? Most likely, correcting it will be a matter of my writing additional code for each case where Tidy isn't producing valid XML, but I'd at least like to stop the script from crashing. So, how can I validate the XML before sending it to the XMPP server?

Dramatization answered 4/11, 2010 at 21:21 Comment(4)
Thanks to Michael Mrozek for the screenshot material! – Dramatization
@TreyE: I'm not sure specifically what you're asking... There isn't a test at the moment. The code just happily takes every HTML message from the chat and parses it to XHTML, and sends that to the XMPP server. I can post relevant code if that helps. – Dramatization
Full source is available in trac or in subversion – Dramatization
I don't know how to create ejabber client by ruby code, plz help me if you can. – Impertinent
D
3

The actual error in this case is your &nbsp;. According to XEP-0071, section 8, point 5:

Section 11.1 of XMPP Core stipulates that character entities other than the five general entities defined in Section 4.6 of the XML specification (i.e., &lt;, &gt;, &amp;, &apos;, and &quot;) MUST NOT be sent over an XML stream. Therefore implementations of XHTML-IM MUST NOT include predefined XHTML 1.0 entities such as &nbsp; -- instead, implementations MUST use the equivalent character references as specified in Section 4.1 of the XML specification (even in non-obvious places such as URIs that are included in the 'href' attribute).

So this issue is about more than just generating well-formed XML, which is a pre-requisite. You'll also want to ensure that you're only using XHTML from the approved set in section 6.

In short, you need to read XEP-0071.

Dole answered 4/11, 2010 at 23:17 Comment(1)
@html_body.gsub!(/&nbsp;/,' ') seems to have made the specific message that was crashing XMPP before stop crashing it now, which is great! However that's clearly just one entity -- I'll need a more robust solution to be sure the XMPP component stays active. Thanks a lot! I'll credit you in the changelog :-) – Dramatization
S
1

Maybe actually converting to XML with Nokogiri will help? You can then re-serialize for the XMPP stream. Also, if you want your stuff to scale a bit and avoid memory bloats, switch to Blather instead of XMPP4r. Also the DSL is pretty awesome!

Sivas answered 4/11, 2010 at 21:33 Comment(2)
Thanks, I'll check out Nokogiri! Can Blather create an XMPP component? That is to say, right now the script creates a new subdomain of my XMPP server, and creates multiple MUC rooms on that subdomain. It's not just a way to connect to an XMPP server using an existing JID... – Dramatization
Ho, yes, specifically if you want to use a component, I would recommand Blather or even Skates. XMPP4R will probably break pretty quickly as traffic increases. – Sivas
N
1

Are you running on *nix? If so, I'd delegate the problem to xmllint, a program that is a part of libxml2. I work with a system that generates xml before sending it over the net; we validate our xml with xmllint, like so:

    command = "xmllint #{temp_file_path} --schema #{schema_file_path} --noout 2>&1"
    output = `#{command}`
    if $? != 0
      temp_dir.keep
      $stderr.puts "Error validating xml: running command #{command.inspect}"
      $stderr.puts output
      exit(1)
    end

You'll need to adapt this to your situation, of course, but the basic idea works well. If you don't have a DTD, leave out the "--schema" bit.

Nymphomania answered 4/11, 2010 at 21:52 Comment(2)
I am running on Linux but I'd prefer the system be OS independent as I know the community has many Windows only users. Also, firing up an external command for every chat message will be awful for performance... have you been in chat recently? ;-) – Dramatization
Unless there's a rubygem for that...? Don't get me wrong, it's a good idea, but... – Dramatization
D
1

Don't use Tidy. Use the HTML5 parser, then dump the DOM it generates to XML. If you can produce a DOM, you can produce well-formed XML from it every time. It will also have the advantage of producing roughly the same DOM that most modern browsers will give you.

Demythologize answered 4/11, 2010 at 22:0 Comment(2)
This sounds extremely promising and I look forward to trying it! – Dramatization
That said, Joe is correct, and additionally, html5lib won't fix the issue he pointed out. – Demythologize

© 2022 - 2024 — McMap. All rights reserved.