1 line
20 KiB
JSON
1 line
20 KiB
JSON
{"repo": "a0938105693-create/phd-level-paper-research-and-analysis-agent", "files": {"README.md": {"b9525bcbe49017d8bd0155b8660fdc449b9d2740": {"t": 1785747165.489313, "n": 11350, "h": "c32d75aef6b4e6b9", "c": "# PhD-Level-Paper-Research-and-Analysis-Agent\n\n[](./LICENSE)\n[](https://www.python.org/)\n[](./schemas/openai_tools.json)\n[](./docs/FULL_PAPER_INGESTION.md)\n\n**ResearchCritic: a full-text, reviewer-grade agent for scientific paper analysis, evidence auditing, and literature synthesis.**\n\nPhD-Level-Paper-Research-and-Analysis-Agent is an open-source research agent for the early stage of scientific work: reading papers, judging evidential strength, identifying methodological weaknesses, and converting accumulated paper analyses into structured literature reviews. The project is built around one methodological premise: useful literature analysis is not summarization. A research assistant must determine whether a paper's experiments, baselines, metrics, ablations, statistics, and limitations actually support the claims being made.\n\nThe agent therefore treats each paper as an object of scientific scrutiny. It establishes a fixed research frame, ingests full paper text whenever available, decomposes the analysis into reviewer-style sections, and produces both a detailed critique and a compact paper memory card for cross-paper synthesis.\n\n## Problem\n\nMost LLM-based paper tools remain shallow at the point where researchers need rigor most. They typically extract abstracts, restate claimed contributions, list experimental results, and generate generic limitations. This is insufficient for PhD-level research work, where the central questions are methodological and evidential:\n\n- Does the paper solve the scientific problem it claims to solve, or only improve a proxy metric?\n- Are the baselines fair with respect to compute, context length, external memory, data access, tools, and tuning?\n- Do ablations establish the necessity of the proposed mechanism?\n- Are reported gains statistically reliable, reproducible, and robust across settings?\n- Are the metrics aligned with the intended research objective?\n- Which conclusions are supported, overstated, or not demonstrated by the evidence?\n- What new research questions become visible after the paper's limitations are audited?\n\nThis project is designed to address these gaps through a full-text, evidence-grounded, sectioned analysis workflow.\n\n## Evidence Audit Matrix\n\nResearchCritic evaluates papers across seven evidence dimensions. These dimensions are designed to force the model away from generic summaries and toward explicit scientific judgment.\n\n| Dimension | Core question |\n|---|---|\n| Scientific problem | What is the real scientific problem, and is it important, testable, and well-posed? |\n| Empirical support | Do the experiments actually support the central claim, or only demonstrate a narrow proxy? |\n| Baseline fairness | Are baselines current and fair with respect to data, tools, context length, external memory, token/compute budget, and tuning? |\n| Mechanism evidence | Do ablations isolate the proposed mechanism, or only show that the full system works? |\n| Metric validity | Do the reported metrics reflect the intended research objective and real deployment constraints? |\n| Statistical reliability | Are results supported by repeated runs, variance, confidence intervals, significance tests, and robustness checks? |\n| Research opportunity | Which limitations, confounders, or failed assumptions create actionable follow-up research? |\n\n## Core Contribution\n\nThe project provides a reproducible agent framework for rigorous scientific literature analysis:\n\n- **Full-text first ingestion**: discovers papers, downloads open-access PDFs, extracts full text, and builds section/evidence digests before formal analysis.\n- **Research-frame conditioning**: requires the user to define the research field, core scientific question, scenario characteristics, and analysis goal before evaluation.\n- **Sectioned reviewer workflow**: generates each major report section with a separate model call, giving experiment auditing, limitation discovery, and research-idea extraction dedicated reasoning budget.\n- **Evidence calibration**: separates author claims, paper evidence, critical judgment, and unknowns caused by missing evidence.\n- **Experimental validity audit**: checks metrics, baselines, ablations, statistics, failure cases, compute/token budget, context/memory fairness, and alternative explanations.\n- **Paper memory cards**: stores structured representations of analyzed papers for later literature survey generation.\n- **Survey synthesis**: organizes accumulated papers by problem, method family, evidence strength, consensus, disagreements, and open research opportunities.\n\n## Produced Research Artifacts\n\nFor each analysis workflow, the project can produce:\n\n- reviewer-style Markdown paper critique reports\n- section maps and evidence-card digests from extracted full text\n- compact PaperMemory cards for later synthesis\n- literature survey reports organized by scientific problem and method family\n- research-gap logs with concrete validation experiments\n- reproducible request payloads and model-output examples for evaluation\n\n## Why It Is Different\n\nThe design is intentionally stricter than conventional paper-summary agents.\n\n| Common paper-summary agents | This project |\n|---|---|\n| Summarize abstracts and claimed contributions | Audits whether claims are supported by full-text evidence |\n| Treat reported metrics as sufficient evidence | Checks metric alignment, proxy failure, and practical meaning |\n| List baselines without questioning fairness | Audits compute, token budget, memory, oracle information, and tuning |\n| Produce generic limitations | Ranks limitations by their effect on the core conclusion |\n| Generate one-shot long reports | Uses sectioned generation with dedicated budget for each reasoning task |\n| Analyze papers independently | Builds paper memory cards for cross-paper synthesis |\n| Stop at \"what the paper says\" | Extracts testable research gaps and validation experiments |\n\nAt the system level, the agent can operate as a complete early-stage research assistant: given a research direction, it can discover recent open papers, download accessible full texts, produce detailed critique reports, retain paper memories, and synthesize a research-area survey.\n\n## System Workflow\n\n```text\nResearch direction\n -> paper discovery from arXiv / open scholarly metadata / top CS venue sources\n -> open PDF download\n -> full-text extraction\n -> section map and evidence-card digest\n -> sectioned LLM critique\n -> reviewer-grade Markdown report\n -> PaperMemory card\n -> cross-paper literature survey\n```\n\nAbstract-only analysis remains available for triage, but formal reports should use full text whenever accessible.\n\n## What the Repository Contains\n\n- Complete Skill/custom-GPT specification\n- Chinese runtime system prompt and report templates\n- OpenAI-compatible tool schemas\n- Python CLI for state management, prompt assembly, paper discovery, PDF download, text extraction, digest generation, and model calls\n- Sectioned generation pipeline for high-quality long reports\n- Evaluation rubric, sample tasks, and model-output examples\n- Documentation for GitHub, Hugging Face, API, and local deployment\n\n## Quick Start\n\nInstall the local toolkit:\n\n```bash\ngit clone https://github.com/YOUR_ORG/PhD-Level-Paper-Research-and-Analysis-Agent.git\ncd PhD-Level-Paper-Research-and-Analysis-Agent\npython3 -m pip install -e .\n```\n\nSet a research frame:\n\n```bash\nresearchcritic --workspace ./workspace frame set \\\n --domain \"LLM agents and autonomous self-improvement\" \\\n --question \"How can language agents improve through reflection without unsupported or regressive strategies?\" \\\n --scenario \"Full-text analysis of recent self-evolving agent papers.\" \\\n --goal \"Critically evaluate whether experimental evidence supports the central claims and identify research gaps.\"\n```\n\nDiscover recent papers and download open PDFs:\n\n```bash\nresearchcritic discover \\\n \"LLM agent self improvement reflection self-evolution\" \\\n --out-dir paper_discovery/agent_self_evolution \\\n --since-year 2024 \\\n --max-results 30 \\\n --venue-profiles ai_ml,nlp\n```\n\nRun full-text sectioned analysis on a discovered paper:\n\n```bash\nresearchcritic --workspace ./workspace deepseek-analyze-discovered \\\n --discovery-dir paper_discovery/agent_self_evolution \\\n --paper-index 1 \\\n --api-key-file /path/to/provider_key \\\n --model deepseek-v4-pro \\\n --thinking \\\n --max-tokens-per-section 4500 \\\n --out reports/p001_fulltext_review.md\n```\n\nThe same pipeline works with any OpenAI-compatible provider that supports chat completions.\n\n## Documentation Map\n\n- Full Skill/custom-GPT specification: [`ResearchCritic_Skill.md`](./ResearchCritic_Skill.md)\n- System prompt: [`prompts/system_prompt.zh.md`](./prompts/system_prompt.zh.md)\n- Single-paper template: [`prompts/single_paper_template.zh.md`](./prompts/single_paper_template.zh.md)\n- Literature survey template: [`prompts/literature_survey_template.zh.md`](./prompts/literature_survey_template.zh.md)\n- Paper discovery: [`docs/PAPER_DISCOVERY.md`](./docs/PAPER_DISCOVERY.md)\n- Full-paper ingestion: [`docs/FULL_PAPER_INGESTION.md`](./docs/FULL_PAPER_INGESTION.md)\n- Sectioned generation: [`docs/SECTIONED_GENERATION.md`](./docs/SECTIONED_GENERATION.md)\n- Evaluation results: [`evals/eval_results.md`](./evals/eval_results.md)\n\n## Example Result\n\nThe repository includes a full-text selected-section evaluation on:\n\n`BenchTrace: A Benchmark for Testing Reflection Ability and Controlled Evolution in LLM Agents`\n\nThe generated report audits full-text evidence including tables, figures, FAR/Score alignment, baseline fairness, statistical reliability, and alternative explanations:\n\n[`evals/model_outputs/benchtrace_deepseek_v4_pro_fulltext_selected.md`](./evals/model_outputs/benchtrace_deepseek_v4_pro_fulltext_selected.md)\n\n## Deployment Targets\n\nThe project can be used as:\n\n- an OpenAI GPT / custom assistant\n- an OpenAI-compatible API agent\n- a local command-line research workflow\n- a Hugging Face repository or Space\n- a Codex-style Skill\n- a component inside a lab literature-review pipeline\n\n## Open-Source Quality\n\nThe repository includes:\n\n- Apache-2.0 license\n- citation metadata\n- contribution guide\n- security policy\n- CI workflow\n- test suite\n- prompt and tool schemas\n- evaluation artifacts\n\nBefore publishing a fork or release, run:\n\n```bash\nPYTHONDONTWRITEBYTECODE=1 PYTHONPATH=src python3 tests/run_stdlib_tests.py\npython3 /path/to/quick_validate.py ./researchcritic\nrg -n \"sk-[A-Za-z0-9]{20,}|DEEPSEEK_API_KEY=|api_key[[:space:]]*=\" . || true\n```\n\n## Privacy and Responsible Use\n\nDo not upload confidential manuscripts, unpublished data, private peer reviews, patient data, or proprietary research materials to external model providers unless the deployment environment is approved for that use. The agent is intended to support research judgment, not replace human peer review or fabricate citations.\n\n## Citation\n\nIf you use this project in academic work or research tooling, cite the repository using [`CITATION.cff`](./CITATION.cff)\n"}, "77da125eb430af147895a44571c568bd73d82f66": {"t": 1785770840.055191, "n": 8689, "h": "ea4255ecc1cb19f1", "c": "# ResearchCritic\n\nRepository: `skill-Paper-research-and-analysis`\n\nResearchCritic is a reviewer-grade AI Skill for scientific literature criticism and evidence-grounded survey generation. It is designed for researchers who need to move beyond paper summaries and perform serious early-stage research planning.\n\nIf you only want the lightweight installable Skill, use the sibling folder:\n\n```text\n/media/mlivs/HDD14T/SKILL/researchcritic\n```\n\nThis repository is the mature open-source toolkit version.\n\nMost paper-analysis assistants answer: \"What does this paper say?\"\n\nResearchCritic asks harder questions:\n\n- What scientific problem is the paper really trying to solve?\n- Do the experiments actually support the central claims?\n- Are the baselines strong, fair, and current?\n- Do the ablations prove the proposed mechanism?\n- Are the metrics aligned with the real research objective?\n- Which conclusions are supported, overstated, or unproven?\n- What hidden limitations create future research opportunities?\n\n## What It Produces\n\n- A Markdown single-paper critical analysis report.\n- A compact paper memory card for future synthesis.\n- A Markdown literature survey generated from previously analyzed papers.\n- Research gaps, experimental recommendations, and thesis/proposal-ready paragraphs.\n\n## Full-Text First\n\nResearchCritic is designed for full-paper critique, not abstract summarization. When an open-access PDF is available, the recommended path is:\n\n1. discover papers from a research direction\n2. download open PDFs\n3. extract full text into `texts/`\n4. build section/evidence digests in `digests/`\n5. run sectioned full-text analysis\n\nAbstract-only analysis is useful for triage, but it must be treated as low-confidence because it cannot verify experiment details, tables, ablations, limitations, or statistical claims.\n\n## Core Workflow\n\nResearchCritic always follows a two-stage workflow:\n\n1. Establish a fixed research frame:\n - research field\n - core scientific question\n - scenario characteristics\n - analysis goal\n2. Route the task:\n - single-paper critical analysis\n - literature survey generation\n - background update\n - optional file save/load actions\n\nThis background-first design prevents shallow generic summaries and lets the model judge whether a paper is actually useful for the user's research problem.\n\n## Quick Start\n\nUse the complete copy-ready specification:\n\n- [`ResearchCritic_Skill.md`](./ResearchCritic_Skill.md)\n\nOr assemble the modular prompt files:\n\n- [`prompts/system_prompt.zh.md`](./prompts/system_prompt.zh.md)\n- [`prompts/single_paper_template.zh.md`](./prompts/single_paper_template.zh.md)\n- [`prompts/literature_survey_template.zh.md`](./prompts/literature_survey_template.zh.md)\n\nFor tool/function calling:\n\n- [`schemas/openai_tools.json`](./schemas/openai_tools.json)\n- [`schemas/gpt_actions.openapi.yaml`](./schemas/gpt_actions.openapi.yaml)\n\n## Install the Local Toolkit\n\nThe repository also includes a lightweight Python toolkit for prompt assembly, state management, arXiv metadata search, safe file operations, and OpenAI-compatible model calls.\n\n```bash\ngit clone https://github.com/YOUR_ORG/skill-Paper-research-and-analysis.git\ncd skill-Paper-research-and-analysis\npython3 -m pip install -e .\n```\n\nIf your system Python is externally managed, create a virtual environment first.\n\nThe toolkit has no mandatory model SDK dependency. It can call OpenAI-compatible chat APIs through the Python standard library. Installing `researchcritic[openai]` enables the official OpenAI SDK path.\n\n## CLI Workflow\n\nSet a research frame:\n\n```bash\nresearchcritic --workspace ./workspace frame set \\\n --domain \"LLM agents and autonomous self-improvement\" \\\n --question \"How can language agents improve through reflection without unsupported or regressive strategies?\" \\\n --scenario \"Early-stage literature review using arXiv metadata and paper text.\" \\\n --goal \"Critically triage papers and identify rigorous research gaps.\"\n```\n\nSearch arXiv without Google Scholar:\n\n```bash\nresearchcritic arxiv-search \"language agents reflexion verbal reinforcement learning\" \\\n --max-results 5 \\\n --out-dir arxiv_results\n```\n\nDiscover recent papers from arXiv plus top CS venue metadata and download open PDFs:\n\n```bash\nresearchcritic discover \\\n \"LLM agent self improvement reflection self-evolution\" \\\n --out-dir paper_discovery/agent_self_evolution \\\n --since-year 2024 \\\n --max-results 30 \\\n --venue-profiles ai_ml,nlp\n```\n\nAnalyze a discovered open-access paper from its extracted full text and digest:\n\n```bash\nresearchcritic --workspace ./workspace deepseek-analyze-discovered \\\n --discovery-dir paper_discovery/agent_self_evolution \\\n --paper-index 1 \\\n --api-key-file /path/to/deepseek_key \\\n --model deepseek-v4-pro \\\n --thinking \\\n --max-tokens-per-section 4500 \\\n --out reports/p001_fulltext_review.md\n```\n\nBuild a section/evidence digest from extracted paper text:\n\n```bash\nresearchcritic paper-digest \\\n --paper paper_text.md \\\n --out digests/paper_digest.md \\\n --json-out digests/paper_digest.json\n```\n\nBuild a one-shot model request:\n\n```bash\nresearchcritic --workspace ./workspace analyze-request \\\n --paper arxiv_results/01_reflexion__language_agents_with_verbal_reinforcement_learning.md \\\n --base-dir . \\\n --out request.json\n```\n\nRun DeepSeek or another OpenAI-compatible provider:\n\n```bash\nresearchcritic deepseek-run \\\n --request-json request.json \\\n --api-key-file /path/to/deepseek_key \\\n --model deepseek-v4-pro \\\n --thinking \\\n --out report.md\n```\n\n## Recommended High-Quality Mode: Sectioned Generation\n\nFor serious research use, prefer sectioned full-text generation over one-shot generation. A full ResearchCritic report is long and intellectually dense; asking a model to produce the entire report in one call can make later sections generic or inconsistent. Sectioned generation calls the model once per report section and passes the previous sections, paper metadata, extracted full text, and optional digest/evidence cards as context.\n\n```bash\nresearchcritic --workspace ./workspace deepseek-sectioned-analysis \\\n --paper arxiv_results/01_reflexion__language_agents_with_verbal_reinforcement_learning.md \\\n --base-dir . \\\n --api-key-file /path/to/deepseek_key \\\n --model deepseek-v4-pro \\\n --thinking \\\n --max-tokens-per-section 3000 \\\n --out reports/reflexion_sectioned.md\n```\n\nWhy this mode is better:\n\n- each section receives a precise objective\n- experiment criticism gets its own model budget\n- limitation discovery is less likely to collapse into generic wording\n- previous conclusions are carried forward as context\n- partial `.part.md` files make long runs recoverable and inspectable\n\nSee [`docs/SECTIONED_GENERATION.md`](./docs/SECTIONED_GENERATION.md) for details.\nSee [`docs/FULL_PAPER_INGESTION.md`](./docs/FULL_PAPER_INGESTION.md) for paper text digestion.\nSee [`docs/PAPER_DISCOVERY.md`](./docs/PAPER_DISCOVERY.md) for automatic paper discovery and open PDF download.\n\n## Deployment Targets\n\nResearchCritic can be deployed as:\n\n- an OpenAI GPT\n- an OpenAI-compatible API assistant\n- a local/self-hosted chat workflow\n- a Hugging Face Space or hosted prompt project\n- a Codex-style Skill\n\nThe prompt-only Skill can be used without Python. The Python toolkit is provided for mature open-source workflows that need reproducible prompt assembly, local state, arXiv discovery, safe file tools, and model-call orchestration.\n\n## Why It Is Different\n\nResearchCritic is intentionally stricter than normal literature-summary prompts. It requires:\n\n- evidence coverage assessment\n- claim-evidence calibration\n- experiment and benchmark criticism\n- baseline fairness checks\n- ablation and mechanism validation\n- limitation severity grading\n- research opportunity extraction\n- multi-paper memory for survey generation\n\nThe goal is to reduce literature-review workload while preserving the intellectual standard expected from a strong human researcher or top-conference reviewer.\n\n## Open-Source Quality Bar\n\nBefore publishing or modifying the Skill, run the prompt-level evaluations in [`evals/`](./evals/):\n\n- background gate behavior\n- abstract-only evidence limitation\n- full-paper critical analysis\n- multi-paper memory\n- survey synthesis\n- save-action confirmation\n\n## Privacy\n\nDo not upload confidential manuscripts, private review assignments, unpublished data, or sensitive research materials to external model providers unless you have permission and the deployment environment satisfies your privacy requirements.\n\n## Citation\n\nIf you use ResearchCritic in academic work or open-source research tooling, cite this repository using [`CITATION.cff`](./CITATION.cff).\n"}}}} |