A Celery-based pipeline for computing and storing discipline-specific search boost factors for the NASA ADS search engine. This pipeline calculates boost factors based on various criteria to prioritize better results in search rankings across different scientific disciplines.
The Boost Pipeline computes boost factors for ADS records based on:
- Refereed status: Refereed publications get higher priority
- Document type: Different document types have configurable ranking-based boost factors
- Recency: Recent publications are prioritized with configurable decay functions
- Collection weights: Discipline-specific weights based on collection rankings
- Final discipline boosts: Combined boost factors for each scientific discipline
The pipeline follows the standard ADS pipeline architecture:
- Celery-based: Asynchronous task processing with multiple queues
- Database storage: PostgreSQL for storing boost factors with comprehensive schema
- Message passing: RabbitMQ for bidirectional communication with Master Pipeline
- Modular design: Separate components for computation, storage, and communication
- Trigger: When Master Pipeline processes any record type (metadata, metrics, etc.), it automatically generates a boost request
- Message Format: JSON message containing:
bibcodeandscix_idat root levelbib_datasection (paper metadata, title, abstract, etc.)metricssection (refereed status, citations)classificationssection (collections/disciplines)
- Transport: Celery message queue system
- Input: Receives boost request messages from Master Pipeline
- Processing: Computes multiple boost factors:
refereed_boost: 1.0 for refereed papers, 0.0 for non-refereeddoctype_boost: Based on document type ranking system (configurable)recency_boost: Time-based decay function (24-month cutoff)
- Collection Weights: Computes discipline-specific weights based on collection rankings
- Final Boosts: Combines basic boosts into discipline-specific final boost scores
- Storage: Saves computed boost factors to PostgreSQL database
- Response: Sends boost factors back to Master Pipeline via Celery
- Integration: Master Pipeline stores boost factors in its
boost_factorsfield
- Refereed Boost: 1.0 for refereed papers, 0.0 for non-refereed
- Document Type Boost: Based on configurable ranking system
- Recency Boost: Reciprocal decay function with 24-month cutoff
- Supports 6 disciplines: astronomy, physics, earth_science, planetary_science, heliophysics, general
- Uses ranking system to assign relevance weights (1.0 to 0.1)
- Handles multiple collections per record by taking maximum weight per discipline
- Combined Boost Factor: Weighted average of basic boost factors
- Discipline Final Boosts:
discipline_weight × combined_boost_factor
- Clone the repository:
git clone <repository-url>
cd ADSBoostPipeline- Install dependencies:
pip install -r requirements.txt- Set up database:
# Create database
createdb boostfactorsdb
# Run migrations
alembic upgrade head- Configure RabbitMQ:
# Create queues
rabbitmqctl add_user test test
rabbitmqctl set_permissions test ".*" ".*" ".*"Edit config.py to configure:
- Database connection (
SQLALCHEMY_URL) - RabbitMQ settings (
CELERY_BROKER_URL) - Boost factor parameters:
DOCTYPE_RANKING: Document type importance rankingsCOLLECTION_RANKINGS: Discipline relevance rankingsBOOST_FACTOR_WEIGHTS: Weights for basic boost factorsRECENCY_BOOST_MULTIPLIER: Time decay parameters
- Logging levels (
LOGGING_LEVEL)
python run.py -b 2023ApJ...123..456T
python run.py -x scix_id_123python run.py -f records.json
python run.py -f records.csvpython run.py -q 2023ApJ...123..456Tpython run.py -e output.csvpython run.pyThe pipeline uses a single, unified batch processing function that handles all bibcode processing scenarios through Celery tasks. Whether processing one bibcode or millions, everything goes through the same efficient, asynchronous Celery task pipeline.
python run.py --process-all --batch-size 500- Purpose: Queries the Master Pipeline's Records database and processes every single record
- Use Case: Initial setup, rebuilding boost factors, or processing entire database
- Features:
- Configurable batch size (default: 100)
- Progress logging every 1000 records
- Automatic record structure creation
- Error handling that continues processing
- Uses Celery tasks for asynchronous processing
python run.py --bibcodes-batch 2023ApJ...123A 2023MNRAS.456B 2023A&A...789C- Purpose: Processes multiple bibcodes provided as command-line arguments
- Use Case: Quick processing of a small set of specific records
- Features:
- Variable number of bibcodes
- Automatic record structure creation
- Individual error handling per record
- Uses Celery tasks for asynchronous processing
python run.py --filename bibcodes.txt --batch-size 200- Purpose: Reads bibcodes from a text file (one per line) and processes them in batches
- Use Case: Processing large lists of bibcodes from external sources
- Features:
- Supports text files with one bibcode per line
- Skips empty lines and comments (lines starting with #)
- Configurable batch size
- Progress logging every 1000 bibcodes
- Uses Celery tasks for asynchronous processing
python run.py -b 2023ApJ...123..456T
python run.py -x scix_id_123- Purpose: Process individual bibcodes or scix_ids
- Implementation: Automatically treated as a batch of 1 for consistency
- Benefits: Same error handling and logging as batch processing
- Uses Celery tasks for asynchronous processing
# Process all records in Master Pipeline database with 500-record batches
python run.py --process-all --batch-size 500
# Process bibcodes from file with 200-record batches
python run.py --filename my_bibcodes.txt --batch-size 200
# Process specific bibcodes from command line
python run.py --bibcodes-batch 2023ApJ...123A 2023MNRAS.456B
# Process single bibcode (automatically uses batch processing)
python run.py -b 2023ApJ...123..456T
# Export existing boost factors to CSV
python run.py --export boost_factors.csv
# Query boost factors for a specific record
python run.py --query 2023ApJ...123AThe pipeline uses a unified approach where all bibcode processing flows through the same Celery task pipeline:
read_bibcodes_from_file(): Extracts bibcodes from text filesprocess_bibcodes_in_batches(): Main batch processing function that submits to Celeryprocess_batch(): Submits records to Celery tasks and manages the task pipelineprocess_all_records_from_master(): Special case for processing entire Master Pipeline database
Celery Task Flow:
- Compute Task:
task_compute_boost_factors.delay(record)- Computes boost factors - Store Task:
task_store_boost_factors.delay(bibcode, scix_id, boost_factors)- Stores in database - Send Task:
task_send_to_master_pipeline.delay(record, boost_factors)- Sends to Master Pipeline
This design eliminates code duplication, ensures consistent behavior across all processing scenarios, and properly leverages Celery for asynchronous, scalable processing whether handling 1 record or 1 million records.
The pipeline uses a comprehensive boost_factors table with the following structure:
id: Primary keybibcode: Bibcode (19 characters)scix_id: SciX ID (19 characters)created: Timestamp
refereed_boost: Refereed status boost factordoctype_boost: Document type boost factorrecency_boost: Recency boost factorboost_factor: Combined weighted average of basic boosts
astronomy_weight: Astronomy discipline weightphysics_weight: Physics discipline weightearth_science_weight: Earth science discipline weightplanetary_science_weight: Planetary science discipline weightheliophysics_weight: Heliophysics discipline weightgeneral_weight: General discipline weight
astronomy_final_boost: Final astronomy boost scorephysics_final_boost: Final physics boost scoreearth_science_final_boost: Final earth science boost scoreplanetary_science_final_boost: Final planetary science boost scoreheliophysics_final_boost: Final heliophysics boost scoregeneral_final_boost: Final general boost score
The pipeline uses multiple Celery queues for different operations:
boost-request: Process incoming boost request messagescompute-boost: Compute boost factors for recordsstore-boost: Store computed boost factors in databasesend-boost-response: Send boost factors back to Master Pipelineexport-boost: Export boost factors to CSV
task_process_boost_request_message: Main entry point for boost requeststask_compute_boost_factors: Compute boost factors for a recordtask_store_boost_factors: Store boost factors in databasetask_send_to_master_pipeline: Send response to Master Pipelinetask_export_boost_factors: Export data to CSV
The Boost Pipeline integrates with the ADS Master Pipeline by:
- Incoming: Receives boost requests via
boost-requestqueue - Processing: Computes boost factors using the simplified algorithm
- Storage: Stores results in PostgreSQL database
- Response: Sends computed boost factors back via
send-boost-responsequeue
- Automatic boost factor computation when records are updated
- Bidirectional communication for real-time processing
- Database integration for persistent storage
- Throughput: Configurable batch processing (default: 100 records/batch)
- Memory usage: Efficient batch processing with configurable sizes
- Database: Optimized queries with indexes on bibcode and scix_id
- Scalability: Multiple Celery workers can process different queues simultaneously
pytest tests/The project follows PEP 8 guidelines.
To add new boost factors:
- Update the
compute_boost_factorsmethod inapp.py - Add corresponding database columns in
models.py - Update the configuration in
config.py - Run database migrations
The pipeline provides comprehensive logging:
- Request processing logs at multiple levels
- Boost factor computation logs with detailed breakdowns
- Database operation logs
- Error tracking with full stack traces
- Progress logging for batch operations
DEBUG: Detailed computation steps and data structuresINFO: Processing progress and successful operationsWARNING: Configuration issues and non-critical problemsERROR: Processing failures and system errors
- Database connection errors: Check
SQLALCHEMY_URLin config - RabbitMQ connection errors: Verify RabbitMQ is running and accessible
- Import errors: Ensure all dependencies are installed
- Batch processing failures: Check batch size configuration and memory limits
Check logs for detailed error information:
tail -f logs/adsboost_pipeline.logThe pipeline validates required configuration parameters:
- Database connection strings
- RabbitMQ broker URLs
- Boost factor computation parameters
- Collection ranking configurations
- Fork the repository
- Create a feature branch
- Make your changes
- Add tests for new functionality
- Update documentation
- Submit a pull request
This project is part of the ADS infrastructure and follows the same licensing terms.
For questions or issues, contact the ADS development team or create an issue in the repository.