Quick answer: Five things decide it, and they run in order. First, what your licenses actually permit, because many publisher contracts prohibit indexing their content into an AI system regardless of what the technology can do. Then retrieval design, where combining keyword and vector search matters more than any model choice. Then document quality, where scanned PDFs quietly destroy accuracy. Then metadata, which is what lets a user narrow to the right slice. Then permissions, enforced at query time rather than at ingest. Most failed deployments fail on the first or the third, not on the AI.
Key takeaways
- Read the license before the architecture. Contract terms override fair use, and some major database agreements prohibit AI use of their content outright.
- Anti-scraping clauses written years ago already cover AI ingestion, because they prohibit indexing by name.
- Hybrid retrieval beats vector search alone by a wide margin, and the gap is largest on exactly the rare terms competitive work depends on.
- Chunking is close to a one-way door. Changing it later re-runs everything downstream.
- Scanned PDFs are where quality quietly dies, and no current OCR tool is considered adequate for building a knowledge base.
- Permissions are applied when someone asks a question, not when the document is loaded, so the entitlement data has to stay in sync.
Start with what you are allowed to index
This is first because it is the step most likely to stop a project after the money is spent, and it is almost never a technology question.
The binding constraint on putting licensed research into an internal AI system is usually the contract, not copyright law. As USC Libraries put it, "contract law and fair use rights are separate sources of authority." A license can forbid things copyright would have permitted, and for subscription research it generally does.
How explicit this has become is worth seeing in the original. EBSCO's 2024 license agreement states that licensees and authorized users "may not use artificial intelligence tools or machine learning technologies with any of the content included in the Databases or Services for any purpose." No carve-out, and notably no distinction between training a model and retrieving for an answer.
That last point undoes an assumption a lot of buyers hold. The training-versus-retrieval distinction is real technically, and vendors lean on it heavily, but it is not reliably reflected in publisher contract language. If the clause says "any purpose," the architecture of your system does not rescue you.
Older clauses catch this too, often by accident. Standard library license language prohibits "the use of robots, spiders, crawlers or other automated downloading programs, algorithms, or devices to continuously search, scrape, extract, or index data or metadata." That was written about crawlers. The word index lands squarely on a retrieval pipeline.
Even the permissive routes are narrower than people assume. Elsevier allows text and data mining by subscribing academic institutions for non-commercial research, through its API rather than bulk download. That is a meaningful allowance, and it is not what a corporate strategy team is doing.
The law here is genuinely unsettled. Thomson Reuters v. Ross Intelligence, where a district court rejected a fair use defense in February 2025, was argued before the Third Circuit in June 2026 and is awaiting a decision expected late in the year. It will be the first federal appellate ruling on the question. Until it lands, the practical answer is unchanged: read the actual agreement for each source, and get your own counsel's view rather than a vendor's. We have written more on why most enterprise AI tools cannot legally read the content you need and on fair use and aggregating web content.
What actually determines whether AI can use the content
Assume the rights question is settled. The next set of decisions has more effect on answer quality than the choice of model, and they are rarely the ones that get debated.
Use hybrid retrieval, because your vocabulary is the problem
Vector search finds things that mean the same. Keyword search finds things that say the same. Competitive and scientific content is full of terms where only the second works: molecule names, ticker symbols, product codes, patent numbers, a competitor's internal project name.
The measured gap is large. In benchmarking published at EMNLP 2024, on the TREC DL 2019 set, keyword retrieval alone scored 30.13 mAP and vector retrieval alone scored 23.99, while the two combined scored 47.14. The authors' explanation is the practical one: dense retrieval "struggles with rare terminologies," while keyword matching "is adept at matching specific terms." If your analysts search for exact identifiers, a vector-only system will lose precisely the queries they care most about.
Treat chunking as a decision you will live with
How documents are split before indexing determines what the system can retrieve. The same EMNLP work found 512-token chunks optimal across its tests, at around 97% on both faithfulness and relevancy, with the trade-off stated plainly: "smaller chunks improve retrieval recall and reduce time but may lack sufficient context."
Microsoft's guidance is blunter about the consequence, and it is the line worth carrying into a planning meeting: "Your chunking approach is a semipermanent choice in your overall solution design." Changing it later means re-processing the corpus and re-running everything downstream. Decide it deliberately, with real documents from your own library rather than a sample.
Structure helps more than tuning. Headers, section headings, tables and captions carry meaning that survives chunking if you preserve them and disappears if you flatten everything to plain text.
Fix the scanned PDFs, or accept the ceiling
This is the most underestimated item on the list. Research published at ICCV 2025 examined how OCR errors propagate through retrieval into generated answers, and distinguished two failure types that behave differently: semantic noise, where characters are misread, and formatting noise, where the text is readable but its structure is scrambled. The second is nastier, because the output looks clean.
The authors' conclusion about the current state of tooling is worth quoting: of the OCR solutions they evaluated, "none is competent for constructing high-quality knowledge bases for RAG systems." That is not a reason to skip scanned material. It is a reason to know which parts of your library are scanned, sample the extraction quality yourself, and set expectations on those sources accordingly.
Which metadata earns its keep
Metadata is where enthusiasm usually outruns value. Teams build elaborate taxonomies that nobody populates, then conclude that tagging does not work.
The test for any field is whether a user would ever filter on it. Three categories usually pass:
- Temporal. Publication and revision dates, so a question can be scoped to the last two quarters rather than everything ever written.
- Categorical. Source type, publisher, document type, confidentiality. This is what separates a licensed analyst report from an internal draft in an answer.
- Hierarchical. Business unit, therapeutic area, product line, project. This is what makes a query mean something specific in a large organization.
Filtering on these narrows the candidate set before ranking, which matters most where the same words carry different meanings in different contexts. Be careful with claims about how much this improves accuracy, though. The platform documentation that recommends metadata filtering does not publish an accuracy figure for it, and neither will we.
Source type is the field most worth insisting on, because it is what lets an answer distinguish "a licensed analyst said this" from "someone on our team drafted this," which is the distinction a reader needs most and the one systems most often lose.
How permissions work once content is in an AI system
The common fear is that loading documents into an AI system flattens access control. In a properly built system it does not, but the mechanism is worth understanding because it has a failure mode.
Enterprise retrieval systems apply permissions at query time. Each document carries a field listing the security principals allowed to see it, and every search is filtered against the identity of the person asking. Microsoft describes the pattern as one that "simulates document-level authorization by using a regular OData filter that includes or excludes a search result based on a string consisting of a security principal."
The implication people miss: that entitlement field is a copy, and copies go stale. If someone changes teams and your index is not updated, the filter enforces yesterday's permissions perfectly. So the operational question is not whether the system supports security trimming. It is how often the entitlement data is refreshed, and what happens between refreshes.
For licensed content there is a second layer. Publisher agreements often limit which users or sites may access a source, and that entitlement has to be represented too, or a compliant contract can be breached by a perfectly functioning search.
How do you know it is working?
Most teams find out from complaints, which is late and unreliable. A small amount of structure is enough.
- Build a question set before you launch. Thirty to fifty real questions from real analysts, with the answer you expect and the document it should come from. This takes an afternoon and becomes the thing you re-run after every change.
- Score retrieval separately from generation. If the right document did not come back, no model will save the answer. Measuring them together hides which half is broken.
- Watch the queries that return nothing. Empty results usually mean a vocabulary gap, a permissions problem, or a source everyone assumes is indexed and is not.
- Re-run the set when anything changes. New sources, a chunking change, a model upgrade. This is the only way to notice a regression before your users do.
Northern Light SinglePoint is one platform built around this problem, describing itself as "one portal" spanning "internal reports, subscriptions, news, dashboards, and SME insights," with a stated 45-day deployment. It was named a Leader in the first-ever 2026 Gartner® Magic Quadrant™ for Competitive and Market Intelligence Platforms. Whether you buy or build, the checklist above is the same. If you are weighing that choice, our post on whether Copilot is enough for market and competitive intelligence covers what general enterprise AI does and does not reach.
Frequently asked questions
Can I put licensed research into an internal AI system?
Only if your license permits it, and many do not. The binding constraint is contract rather than copyright, because license terms can override fair use. EBSCO's 2024 agreement, for example, states that users may not use artificial intelligence tools or machine learning technologies with its content for any purpose. Older anti-scraping clauses that prohibit automated indexing also apply to retrieval pipelines even though they predate AI. Read each agreement and get your own counsel's reading rather than relying on a vendor's.
Does it matter whether AI is trained on the content or just retrieves it?
Technically yes, contractually often not. Training a model on content and retrieving passages at query time are genuinely different operations, and vendors lean on that distinction. But publisher contract language frequently does not make it. A clause prohibiting use of AI tools with the content for any purpose covers retrieval as squarely as training, so the architecture of your system does not resolve the licensing question.
What is the biggest technical factor in retrieval quality?
Combining keyword and vector search rather than relying on either alone. Benchmarking published at EMNLP 2024 found that on the TREC DL 2019 set, keyword retrieval scored 30.13 mAP and vector retrieval 23.99, while hybrid retrieval scored 47.14. The reason matters for competitive work specifically: vector search struggles with rare terminology, which is exactly what molecule names, ticker symbols and product codes are.
How should I chunk documents for AI retrieval?
Around 512 tokens performed best in EMNLP 2024 benchmarking, but treat any figure as a starting point to test on your own documents. The more important point is that chunking is close to a one-way door. Microsoft describes it as a semipermanent choice, because changing it later means reprocessing the corpus and re-running everything downstream. Preserve document structure such as headings and tables, which carries meaning that plain-text extraction destroys.
Do scanned PDFs work in an AI system?
They work poorly, and the damage is easy to miss. Research presented at ICCV 2025 traced how OCR errors cascade through retrieval into generated answers, distinguishing semantic noise, where characters are misread, from formatting noise, where text is legible but its structure is scrambled. The authors concluded that of the OCR solutions evaluated, none was competent for building high-quality knowledge bases. Identify which parts of your library are scanned and sample the extraction quality before trusting answers drawn from them.
Will AI expose documents people should not see?
Not in a properly built system, which filters every query against the identity of the person asking rather than relying on what was loaded. Each document carries the security principals permitted to see it and searches are filtered accordingly. The real risk is staleness: that entitlement data is a copy of your authorization system, so the practical questions are how often it refreshes and what happens in between. Licensed content adds a second layer, since publisher agreements often limit which users may access a source.
The bottom line
The pattern across failed deployments is consistent, and it is rarely the model. It is a library nobody checked the rights on, a corpus half of which is scanned, a vector-only index that cannot find a product code, and no test set to notice any of it.
None of that is glamorous work and all of it is tractable. Start with the licenses, because that is the step that can stop everything later. Then make the content retrievable, then make it filterable, then make it governed, then measure whether it is working. The AI is the easy part. What it is allowed to read, and how well, is the whole job.
Want the deeper technical case for grounding enterprise AI in licensed and internal sources? Download Northern Light's technical whitepaper, The Architecture Behind Northern Light AI.
Sources: USC Libraries, Copyright and Licensed Materials, 2026. US Army War College Library, Database restrictions on AI tools, 2026, quoting the EBSCO 2024 License Agreement. Elsevier, Text and data mining policy. Baker Botts, Third Circuit hears oral argument in Thomson Reuters v. Ross Intelligence, 2026 (decision pending). Wang X et al., Searching for Best Practices in Retrieval-Augmented Generation, EMNLP 2024. Zhang J et al., OCR Hinders RAG, ICCV 2025. Microsoft, Develop a RAG solution, chunking phase, 2026; Security trimming for Azure AI Search, 2026. AWS, Metadata filtering in Bedrock Knowledge Bases, 2024. This article describes general practice and is not legal advice.
Gartner and Magic Quadrant are registered trademarks and service marks of Gartner, Inc. and/or its affiliates in the U.S. and internationally and are used herein with permission. All rights reserved. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation.
