Search ArXiv Papers Without XML Parsing Headaches

If you've ever tried to search ArXiv programmatically, you've probably stared at XML responses wondering why something so simple has to be so complicated. Those nested namespaces, the Atom 1.0 format, and the constant worry about whether your XML parser will handle the next edge case correctly.

We've all been there. You just want to search for papers about "quantum computing" and get back a nice JSON array. Instead, you're debugging xmlns attributes at 2 AM.

Understanding the ArXiv XML Challenge

ArXiv's API returns results in Atom 1.0 format, which made sense in 2007 when the API was designed. Today, it creates unnecessary complexity for developers who expect JSON responses from modern APIs.

Here's what a typical ArXiv API response looks like:

<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <link xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/"
        rel="self" type="application/atom+xml"/>
  <title xmlns="http://www.w3.org/2005/Atom">ArXiv Query</title>
  <entry>
    <id>http://arxiv.org/abs/2301.00001v1</id>
    <updated>2023-01-01T00:00:00Z</updated>
    <published>2023-01-01T00:00:00Z</published>
    <title>Quantum Computing Applications</title>
    <author>
      <name>John Doe</name>
    </author>
  </entry>
</feed>

The multiple namespaces and nested structure make parsing error-prone. Miss one namespace declaration and your entire parser breaks.

Method 1: Traditional ArXiv API with XML Parsing

Let's look at the traditional approach using the ArXiv API directly:

import { XMLParser } from 'fast-xml-parser';

async function searchArxivTraditional(query: string) {
  const url = `http://export.arxiv.org/api/query?search_query=${encodeURIComponent(query)}&max_results=10`;

  const response = await fetch(url);
  const xmlText = await response.text();

  const parser = new XMLParser({
    ignoreAttributes: false,
    removeNSPrefix: false
  });

  const result = parser.parse(xmlText);
  const entries = result.feed.entry || [];

  // Transform to clean JSON
  return (Array.isArray(entries) ? entries : [entries]).map(entry => ({
    id: entry.id,
    title: entry.title,
    authors: Array.isArray(entry.author)
      ? entry.author.map(a => a.name)
      : [entry.author?.name],
    published: entry.published,
    summary: entry.summary
  }));
}

// Usage
const papers = await searchArxivTraditional("quantum computing");
console.log(papers);

Pros:

• Direct access to ArXiv API
• No third-party API keys needed
• Free to use

Cons:

• Complex XML parsing logic
• Namespace handling is fragile
• No built-in error handling for malformed XML
• Must handle rate limiting manually (3 second delays)

Method 2: Python ArXiv Library

The official Python library simplifies things somewhat:

import arxiv
import json

def search_arxiv_python(query):
    search = arxiv.Search(
        query=query,
        max_results=10,
        sort_by=arxiv.SortCriterion.SubmittedDate
    )

    results = []
    for result in search.results():
        results.append({
            "id": result.entry_id,
            "title": result.title,
            "authors": [author.name for author in result.authors],
            "published": result.published.isoformat(),
            "summary": result.summary
        })

    return json.dumps(results)

# Usage
papers = search_arxiv_python("quantum computing")
print(papers)

Pros:

• Official library handles XML parsing
• Cleaner code than manual parsing
• Automatic retry logic

Cons:

• Python-only solution
• Still requires JSON transformation
• Not suitable for JavaScript/TypeScript projects

Method 3: PDF Vector Academic Search API

PDF Vector provides a modern alternative with native JSON responses:

async function searchArxivWithPDFVector(query: string) {
  const response = await fetch('https://www.pdfvector.com/v1/api/academic-search', {
    method: 'POST',
    headers: {
      'Authorization': 'Bearer pdfvector_xxx',
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      query: query,
      providers: ['arxiv'],
      limit: 10,
      fields: ['title', 'authors', 'year', 'abstract', 'pdfURL']
    })
  });

  const data = await response.json();
  return data.results;
}

// Usage
const papers = await searchArxivWithPDFVector("quantum computing");
console.log(papers);

// Clean JSON response:
// [\
//   {\
//     "title": "Quantum Computing Applications in Machine Learning",\
//     "authors": [\
//       { "name": "John Doe" },\
//       { "name": "Jane Smith" }\
//     ],\
//     "year": 2023,\
//     "abstract": "We explore the intersection of quantum computing...",\
//     "pdfURL": "https://arxiv.org/pdf/2301.00001.pdf"\
//   }\
// ]

Want to search across multiple databases? Just add more providers:

const results = await fetch('https://www.pdfvector.com/v1/api/academic-search', {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer pdfvector_xxx',
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    query: "quantum computing applications",
    providers: ['arxiv', 'semantic-scholar', 'pubmed'],
    limit: 20
  })
});

Pros:

• Clean JSON responses with no XML parsing needed
• Search multiple academic databases simultaneously
• Consistent data structure across all providers
• No rate limiting issues for reasonable usage
• TypeScript SDK available

Cons:

• Requires API key (free tier available with 100 credits/month)
• Credit-based system (2 credit per search)

Comparing the Approaches

Aspect	ArXiv Direct	Python Library	PDF Vector
Response Format	XML	Python objects	JSON
Parsing Complexity	High	Medium	None
Error Handling	Manual	Built-in	Built-in
Multiple Databases	No	No	Yes
Setup Time	1 day	5 minutes	5 minutes

Making the Right Decision

Use ArXiv Direct API when:

• You're building a one-off script or prototype
• You have existing XML parsing infrastructure
• You need unlimited free queries
• You're comfortable handling XML namespaces
• Rate limiting won't affect your use case

Use Python arxiv library when:

• You're already in a Python environment
• You need the official implementation
• You want built-in error handling
• You can work within the rate limits
• You prefer Python objects over raw XML

Use PDF Vector when:

• You want clean JSON responses without XML parsing
• You need to search multiple academic databases
• You value development speed over free access
• You're building a production application
• You need consistent data structure across providers