metadata fix

packages/markdown-assistant/README.md

@@ -1,6 +1,6 @@
-# Markdown Processor
+# Markdown Assistant
 
-This package provides scripts to process markdown files using the OpenAI API. It includes functionalities to format markdown, generate metadata, and ensure accessibility of images with alt text. The package supports batch processing to handle multiple requests efficiently.
+This project provides tools to process and format markdown files using OpenAI's GPT models. The primary script, `index.js`, allows for both batch and individual file processing, with options for custom instructions and output handling.
 
 ## Prerequisites
 
@@ -10,159 +10,114 @@ This package provides scripts to process markdown files using the OpenAI API. It
 
 ## Installation
 
-1. Clone the repository:
-
-   ```
-   git clone <repository_url>
-   ```
-
-2. Navigate to the project directory and install dependencies:
-
+1. Clone the repository.
+2. Install dependencies using npm or yarn:
+
    ```
-   cd markdown-processor
    npm install
    ```
 
-3. Create a `.env` file in the root directory with your OpenAI API key:
+   or
 
    ```
-   OPENAI_API_KEY=your_openai_api_key
+   yarn install
    ```
 
-## Usage
-
-### Format Markdown
-
-Format a markdown file or all markdown files in a directory using OpenAI API.
-
-#### Command
-
+3. Create a `.env` file in the root directory and add your OpenAI API key:
+
    ```
-   node index.js --file path/to/markdown.md [--instruction "Custom instruction"] [--output path/to/output.md] [--batch]
+   OPENAI_API_KEY=your_openai_api_key_here
    ```
 
-#### Options
-
-- `--file, -f`: Path to the markdown file or directory containing markdown files (required)
-- `--instruction, -i`: Custom instruction for formatting the markdown
-- `--output, -o`: Path to the output file. If not provided, the output file will be the input filename with `.out` added before the extension.
-- `--batch, -b`: Create a batch file and use output if present
-
-### Generate Metadata
-
-Generate JSON-LD metadata for a markdown file or all markdown files in a directory.
-
-#### Command
-
-   ```
-   node metadata.js --file path/to/markdown.md [--instruction "Custom instruction"] [--output path/to/output.md] [--embed] [--batch]
-   ```
-
-#### Options
-
-- `--file, -f`: Path to the markdown file or directory containing markdown files (required)
-- `--instruction, -i`: Custom instruction for generating metadata
-- `--output, -o`: Path to the output file. If not provided, the output file will be the input filename with `.out` added before the extension.
-- `--embed, -e`: Embed JSON-LD metadata into the original markdown
-- `--batch, -b`: Create a batch file and use output if present
-
-### Ensure Image Accessibility
-
-Add or overwrite alt text for images in a markdown file or all markdown files in a directory based on accessibility guidelines.
-
-#### Command
-
-   ```
-   node a11y-images-alt.js --file path/to/markdown.md [--output path/to/output.md] [--overwrite] [--batch]
-   ```
+## Usage
 
-#### Options
+### Command Line Options
 
-- `--file, -f`: Path to the markdown file or directory containing markdown files (required)
-- `--output, -o`: Path to the output file. If not provided, the output file will be the input filename with `.out` added before the extension.
-- `--overwrite, -ow`: Overwrite existing alt attributes
-- `--batch, -b`: Create a batch file and use output if present
+- `--file, -f`: Path to the markdown file or directory (required).
+- `--instruction, -i`: Instruction to process the markdown (default: 'Please format this markdown correctly.').
+- `--output, -o`: Output file path (optional).
+- `--batch, -b`: Create a batch file and use output if present (default: false).
+- `--model, -m`: Model to use for processing (default: 'gpt-4o-mini').
+- `--overwrite, -w`: Overwrite the input file with the processed content (default: false).
 
-### Batch Processor
+### Examples
 
-Process batch files created by the above scripts. The batch processor will submit new jobs, check for completed jobs, output the results, and clean up the batch files of completed jobs.
+#### Process a Single File
 
-#### Command
+To process a single markdown file and output the result to a new file:
 
-   ```
-   node batch_processor.js --input path/to/batch_file_or_directory
-   ```
+```
+node index.js --file path/to/file.md --output path/to/output.md
+```
 
-#### Options
+#### Process a Directory of Files
 
-- `--input, -i`: Path to the batch file or directory containing batch files (required)
+To process all markdown files in a directory:
 
-## Example Workflows
+```
+node index.js --file path/to/directory --batch
+```
 
-### Formatting Markdown and Using Batch Output
+#### Overwrite the Original File
 
-1. Create a batch file for formatting markdown:
+To overwrite the original markdown file with the processed content:
 
-   ```
-   node index.js --file path/to/markdown.md --batch
-   ```
+```
+node index.js --file path/to/file.md --overwrite
+```
 
-2. Process the batch file:
+#### Custom Instruction
 
-   ```
-   node batch_processor.js --input path/to/markdown.md.batch.jsonl
-   ```
+To provide a custom instruction for processing:
 
-3. The next time you run the script with the `--batch` option, it will use the batch output if present:
+```
+node index.js --file path/to/file.md --instruction "Your custom instruction here"
+```
 
-   ```
-   node index.js --file path/to/markdown.md --batch
-   ```
+### Batch Processing Scripts
 
-### Generating Metadata and Using Batch Output
+#### `batch_processor_submit.js`
 
-1. Create a batch file for generating metadata:
+This script is used to submit batch jobs to OpenAI.
 
-   ```
-   node metadata.js --file path/to/markdown.md --batch
-   ```
+```
+node batch_processor_submit.js --input path/to/batch_file.jsonl
+```
 
-2. Process the batch file:
+#### `batch_processor_fetch.js`
 
-   ```
-   node batch_processor.js --input path/to/markdown.md.batch.jsonl
-   ```
+This script is used to fetch results from previously submitted batch jobs.
 
-3. The next time you run the script with the `--batch` option, it will use the batch output if present:
+```
+node batch_processor_fetch.js --input path/to/batch_file.jsonl
+```
 
-   ```
-   node metadata.js --file path/to/markdown.md --batch
-   ```
+### Metadata Processing
 
-### Ensuring Image Accessibility and Using Batch Output
+#### `metadata.js`
 
-1. Create a batch file for ensuring image accessibility:
+This script processes metadata for markdown files. It extracts metadata and appends it to the markdown files.
 
-   ```
-   node a11y-images-alt.js --file path/to/markdown.md --batch
-   ```
+```
+node metadata.js --file path/to/markdown_file.md
+```
 
-2. Process the batch file:
+### Accessibility Image Alt Text
 
-   ```
-   node batch_processor.js --input path/to/markdown.md.batch.jsonl
-   ```
+#### `a11y-images-alt.js`
 
-3. The next time you run the script with the `--batch` option, it will use the batch output if present:
+This script generates alt text for images in markdown files to improve accessibility.
 
-   ```
-   node a11y-images-alt.js --file path/to/markdown.md --batch
-   ```
+```
+node a11y-images-alt.js --file path/to/markdown_file.md
+```
 
-## Contributing
+## Additional Information
 
-Contributions are welcome! Please open an issue or submit a pull request on GitHub.
+- The script processes up to 10 files concurrently to avoid rate limits.
+- It will only process files containing the comment `<!-- llmformatted -->` unless the `--overwrite` flag is set.
+- If rate limits are encountered, the script will pause and retry after a minute.
 
 ## License
 
-This project is licensed under the MIT License.
+This project is licensed under the MIT License. See the LICENSE file for details.

packages/markdown-assistant/index.js

@@ -5,6 +5,8 @@ import fs from 'fs/promises';
 import path from 'path';
 
 const DEFAULT_MODEL = 'gpt-4o-mini';
+const CHUNK_SIZE = 10;
+const LLM_COMMENT = '<!-- llmformatted -->';
 
 async function formatMarkdown(options) {
   if (!await loadEnv() || !process.env.OPENAI_API_KEY) {
@@ -30,11 +32,20 @@ async function formatMarkdown(options) {
     const batchFilePath = `${filePath}.batch.jsonl`;
     const outputFilePath = options.overwrite ? filePath : (options.output || filePath.replace(/\.md$/, '.out.md'));
 
+    // Check if the file has already been formatted if not overwriting
+    if (!options.overwrite) {
+      const content = await fs.readFile(filePath, 'utf8');
+      if (content.includes(LLM_COMMENT)) {
+        console.log(`Skipping already formatted file: ${filePath}`);
+        return;
+      }
+    }
+
     if (options.batch) {
       const batchOutput = await readBatchOutput(batchFilePath);
       if (batchOutput) {
-        console.log(batchOutput);
-        await fs.writeFile(outputFilePath, batchOutput);
+        const outputContent = `${LLM_COMMENT}\n${batchOutput}`;
+        await fs.writeFile(outputFilePath, outputContent);
         console.log(`Output written to ${outputFilePath}`);
       } else {
         await createBatchFile({
@@ -46,26 +57,44 @@ async function formatMarkdown(options) {
         });
       }
     } else {
-      const formattedContent = await processMarkdown({
-        apiKey: process.env.OPENAI_API_KEY,
-        filePath,
-        instruction,
-        model: options.model
-      });
+      let formattedContent;
+      while (true) {
+        try {
+          formattedContent = await processMarkdown({
+            apiKey: process.env.OPENAI_API_KEY,
+            filePath,
+            instruction,
+            model: options.model
+          });
+          break; // Exit the loop if successful
+        } catch (error) {
+          if (error.code === 'rate_limit_exceeded') {
+            console.log('Rate limit exceeded. Waiting for 60 seconds before retrying...');
+            await new Promise(resolve => setTimeout(resolve, 60000));
+          } else {
+            throw error; // Re-throw if not a rate limit error
+          }
+        }
+      }
 
-      await fs.writeFile(outputFilePath, formattedContent);
+      const outputContent = `${LLM_COMMENT}\n${formattedContent}`;
+      await fs.writeFile(outputFilePath, outputContent);
       console.log(`Output written to ${outputFilePath}`);
     }
   }
 
+  async function processFilesInChunks(files) {
+    for (let i = 0; i < files.length; i += CHUNK_SIZE) {
+      const chunk = files.slice(i, i + CHUNK_SIZE);
+      await Promise.all(chunk.map(file => processFile(file)));
+    }
+  }
+
   const stats = await fs.stat(options.file);
   if (stats.isDirectory()) {
     const files = await fs.readdir(options.file);
-    for (const file of files) {
-      if (file.endsWith('.md')) {
-        await processFile(path.join(options.file, file));
-      }
-    }
+    const markdownFiles = files.filter(file => file.endsWith('.md')).map(file => path.join(options.file, file));
+    await processFilesInChunks(markdownFiles);
   } else {
     await processFile(options.file);
   }

packages/markdown-assistant/meta-test/1817-05-05.md

@@ -0,0 +1,42 @@
+<!-- llmformatted -->
+# Board of Visitors minutes (1817) May 5, 1817
+
+## Board of Visitors Minutes
+
+University of Virginia, Board of Visitors\
+May 5, 1817\
+University of Virginia Library\
+Charlottesville, Virginia\
+2006
+
+## Board of Visitors Minutes
+
+### Public Minutes
+
+4 pages
+
+The official minutes contain salary information. Access to minutes containing salary information will be granted only under the procedures of the Virginia Freedom of Information Act (Code of Virginia-2.1-340 ff.) or if permission is granted by the Secretary of the Board. Researchers interested in obtaining access to these records should direct a request in writing to the Office of the Legal Adviser, Madison Hall, University of Virginia. These public minutes have had salary information deleted and bear no restrictions.
+
+Board of Visitors' minutes, Accession # RG-1/1/, Special Collections Dept., University of Virginia, Charlottesville, Va.
+
+Minutes of meetings of the university's governing body which is the legal owner and operator of the institution. Recorded in the minutes are appointments and other personnel decisions concerning faculty, decisions about construction of facilities, gifts to the university, and all other board actions, decisions and resolutions.
+
+At a meeting of the Visitors of the Central College held at Charlottesville on the 5th day of May 1817, on a call by three members, to wit, John Hartwell Cocke, Joseph C. Cabell & Th Jefferson, present James Monroe, James Madison, John H. Cocke, and Th Jefferson.
+
+The records of the trustees of the Albemarle academy, in lieu of which the Central College is established, were received from their Secretary by the hands of Alexr Garrett one of the sd trustees. Resolved that Valentine W. Southall be appointed Secretary to the board, and that the sd records be delivered to him.
+
+The board proceeded to the appointment of a Proctor, and the said Alexander Garrett was appointed, with a request that he will act as Treasurer also until a special appointment can be made.
+
+The board being informed that at a meeting which had been proposed for the 8th day of April last at Charlottesville and at which the three members only who called this present meeting had attended, the sd members had visited & examined the different sites for the college within a convenient distance around Charlottesville had deemed the one offered them by John Perry about a mile above the town, to be the most suitable, and offered on the most reasonable terms, and had provisionally authorised a purchase of certain parcels thereof for the site of the sd college & its appendages, and the members now present having themselves proceeded to the sd grounds, examined them, and considered the terms of the sd provisional purchase, do now approve of the sd grounds as a site for the sd college & its appendages and of the terms of purchase, which they hereby re-afirm & ratify, and they accordingly authorise their Proctor above named to proceed to a regular conveyance thereof to himself & his successors in trust for the sd College.
+
+The act establishing the Central college having transferred to the same all the rights and claims existing in the Albemarle academy & its trustees, and having in aid of the subscriptions & donations obtained or to be obtained & of the proceeds of the lottery authorised by law specially empowered this college by its proper officers to demand and receive the monies which arose from the sales of the Glebe lands of the parishes of Saint Ann & Fredericksville, or such part thereof as belongs to the county of Albemarle or its citizens, in whatever hands they may be, to be employed for the purposes of this college. Ordered that the Proctor enquire into the state of the said property & report the same to this board; and that in the mean time he be authorised to demand and receive so much of the said monies as may be requisite to pay for the lands purchased from the sd John Perry, and to make payment accordingly.
+
+On view of a plan presented to the trustees of the Albemarle academy for erecting a distinct pavilion or building for each separate professorship, and for arranging these around a square, each pavilion containing a schoolroom & two apartments for the accommodation of the Professor with other reasonable conveniences the board determines that one of those pavilions shall now be erected; and they request the Proctor, so soon as the funds are at his command to agree with proper workmen for the building of one, of stone or brick below ground, & of brick above, of substantial work, of regular architecture, well executed, and to be completed if possible during the ensuing summer & winter: that the lot for the sd pavilions be delineated on the ground of the breadth of    feet with two parallel sides of indefinite length, and that the pavilion first to be erected be placed on one of the lines so delineated, with its floor in such degree of elevation from the ground as may correspond with the regular inclined plane to which it may admit of being reduced hereafter.
+
+And it is further resolved that so far as the funds may admit, the Proctor be requested to proceed to the erection of dormitories for the students adjacent to the said pavilion, not exceeding ten on each side, of brick, & of regular architecture according to the same plan proposed.
+
+The board proceeding to consider the plan of a lottery prepared by the trustees of the Albemarle academy approve of the same, and resolve that it be carried into execution and without delay by the Proctor and by such agents as he shall appoint, and that the monies to be received for tickets by those entrusted with the sale of them, be from time to time & at short periods paid into the hands of the proctor and by him deposited in the bank of Virginia in Richmond, with which bank it is thought expedient that an account should be opened with him in trust for the Central College.
+
+Resolved that a subscription paper be prepared, and placed in such hands as the Proctor shall deem will be most likely to promote it with energy and success in which shall be different columns, to wit one for those who may prefer giving a donation in gross, another for those who may be willing to give a certain sum annually for the term of four years; and a third for donations in any other form; and that the monies subscribed be disposed of as they are received by the Proctor, in the manner above prescribed for those received on the lottery.
+
+Resolved that Thomas Jefferson and John H. Cocke be a Committee on the part of the Visitors with authority jointly or severally to advise and sanction all plans and the application of monies for executing them, which may be within the purview and functions of the Proctor for the time being.