Issues

• [later] Develop a storage solution for API keys (initially as environment variables, later as a Key Management System).

  peerbench/peerBench#24 · Filter by author saifmir700 opened on Mar 19, 2025

• Implement rate limiting so the next prompt is sent only after the first is completed with all providers or timed out

  peerbench/peerBench#23 · Filter by author saifmir700 opened on Mar 19, 2025

• Script that lets validators send prompts to all providers simultaneously.

  peerbench/peerBench#22 · Filter by author saifmir700 opened on Mar 19, 2025

• Py SDK functions to create CID from JSON, Dict (convert to JSON first), and raw file

  peerbench/peerBench#21 · Filter by author saifmir700 opened on Mar 19, 2025

• SDK function to validate signatures on Aggregate Metrics Evaluation JSON, Response SCore Audit JSON

  peerbench/peerBench#20 · Filter by author saifmir700 opened on Mar 19, 2025

• Build an SDK framework to allow AI model providers to add plugins for their API. SEE DETAILS

  peerbench/peerBench#19 · Filter by author saifmir700 opened on Mar 19, 2025

• Develop methods for AI researchers and users to spot-check validator responses. | Implement a mechanism to reprompt AI providers and verify response quality.

  peerbench/peerBench#18 · Filter by author saifmir700 opened on Mar 19, 2025

• • Implement did:key or did:pkh for validator identities. | Develop a centralized resolver for metadata linked to DIDs. | Ensure validators use a consistent did:string as their immutable ID | [Later] Support did:plc or similar for key rotation.

  peerbench/peerBench#17 · Filter by author saifmir700 opened on Mar 19, 2025

• • Implement did:key or did:pkh for validator identities. • Develop a centralized resolver for metadata linked to DIDs. • Ensure validators use a consistent did:string as their immutable ID. • [Later] Support did:plc or similar for key rotation.

  peerbench/peerBench#16 · Filter by author saifmir700 opened on Mar 19, 2025

• dEval #15

  peerbench/peerBench#15 · Filter by author saifmir700 opened on Mar 19, 2025

• Quick analyze of papersWithcodeData to find which AI test dataset to see which are the most frequently used for bencharmking

  peerbench/peerBench#14 · Filter by author reisepass opened on Mar 19, 2025
  1

• Derive JSON schema from sample data

  peerbench/peerBench#13 · Filter by author reisepass opened on Mar 19, 2025

• Take some real world AI benchmark results from your work and try to convert them by hand into our Evaluation Schema and Score Audit Schema

  peerbench/peerBench#12 · Filter by author reisepass opened on Mar 19, 2025

• find more colabs

  peerbench/peerBench#11 · Filter by author reisepass opened on Mar 19, 2025